Writing My Own Database From Scratch
Vložit
- čas přidán 12. 06. 2024
- First episode of the #mkown series.
In this video I write my own SQL database from scratch without using any libraries or third party dependencies. Only the standard library provided by the chosen programming language is allowed (Rust in this case). The end result is a basic ACID compliant database that can process only one transaction at a time, something similar to the first versions of SQLite from the early 2000s, but with less features (and probably much worse performance).
The goal of these videos is to provide a high level overview of how such complicated systems work internally, not explaining all the code in detail as that would require dozens of hours of content.
🌐 LINKS
Project Repository: github.com/antoniosarosi/mkdb
✉️ CONTACT INFO
Business Email: business@antoniosarosi.io
Contact Email: sarosiantonio@gmail.com
Twitter: / antoniosarosi
Instagram: / antoniosarosi
🎵 MUSIC
🕖 Time Lapses
/ temporal-t-lpa-once-up...
• Neovaii - The Maze
🔉 Background
/ intro-falling-out
/ silhouette
/ cerulean-skyway
/ lifted
• Crossroad
• [FREE] Chill Type Beat...
• Sine Ira
• ❰Chillstep❱ Sappheiros...
📖 CHAPTERS
00:00 What is Mkown?
00:37 What Do We Know About Databases?
02:18 Research Time Lapse
02:53 Initial Ideas
03:04 Naive Storage Format
05:31 Why Binary Trees Are Not Enough For Indexes
07:46 Why B-Trees Are Better Than Binary Trees
12:21 Software Architecture
13:17 Dev Time Lapse
14:13 Project Demo
18:23 Database Internals
19:24 File System Structure
20:31 Slotted Pages
23:45 Rows, Tables And Indexes
26:50 Overflow Pages
28:30 Sophisticated BTree Balancing Algorithm
29:53 Parsing and AST (Abstract Syntax Tree)
31:54 Query Planner
34:23 K-Way External Merge Sort
36:50 Cache Replacement Algorithm
38:19 Transactions: Commit & Rollback
39:27 Network Protocol
39:56 Final Thoughts
🏷️ HASHTAGS
#programming
#computerscience - Věda a technologie
It's been 4 weeks since you released your video. Man, it is unfair that you have only 1.8k views. If I stumbled upon this video, then you will find your viewers. Keep doing this!!
That's how CZcams works at the beginning, it recommends videos slowly until it gathers enough data to determine whether the video is worth recommending to the masses. I have another channel with over 160K subs and that's what happened with that one. It's gonna be a hard journey, but I'm pretty sure it'll work out in the end. Anyway, I'm glad people like you are finding my videos. Stay tuned for more content like this!
This is an amazingly edited and insightful video!
Thanks!
cool stuff. thats how i learn stuff also. i always teardown whats going on under the hood.
Yeah that's the way to go, especially with software like this. Feels impossible to understand at first but if you persist you end up figuring it out.
love this content man
Thanks man
This was a great video highlighting the overall design of db internal working.
I am currently researching about databases to implement one of my own as a side project.
Totally loved this video. Thanks.
I saw one of your comments about using Rust. Would you choose another language if you were to build another db? If yes, which one?
I'd like to try Zig for a project like this because it's more straightforward when it comes to low level code, it's similar to C but it's a more modern language, you don't have to write basic data structures like lists yourself. I've heard about other systems languages like Jakt or Nim but never tried those myself, Zig I've used for some Advent of Code problems and I liked it. You could also use C++ if you like it but no idea about that one, I only know C. Go is also suitable, it's not as low level, it has a garbage collector so you don't have to deal with memory yourself.
But don't get me wrong Rust works as well, it's just that you have to fight against the compiler so many times, it requires a lot of refactoring. Good news is that once you refactor and it compiles, most of the time everything just works, bad news is that you have to refactor more often.
And then there's the whole "unsafe abstraction" discussion which in theory you should be able to build a "safe" API on top of unsafe code, but then you still have to please the compiler to make that safe abstraction work.
So I wonder if all the time spent on making sure the Rust compiler is happy could be more efficiently spent on debugging seg faults in other languages.
If you've never used Rust I definitely recommend it, because it's going to teach you a lot of stuff about low level details, but if you have already used Rust for other similar projects I would try something else for databases in particular.
For higher level code that doesn't have to work with bits and bytes I'd still use Rust, because I like the high level abstractions like closures, iterators, etc, which usually you only get in high level languages like Python.
Long story short, use Rust if you don't have a lot of experience with it, but if you are already at the point where you can predict "can't borrow as mutable because also borrowed as immutable" errors without even compiling, then you know enough Rust, try something else 😂.
@@tony_saro why not java?
You can use anything really even Python if you want but I wouldn't consider languages that require an interpreter or VM suitable for systems programming. It feels slow and bloated compared to a single machine code binary.
I'm also writing my own database manager too, it's a pain and a mess, 😂 but more than my code, I'm sticking with the concepts I'm learning. As soon as the semester ends I plan to rewrite my code, if possible in Rust now that my current project is in C++, thank you very much for your content. 🎉
It's definitely a pain, this is the hardest project I've ever worked on, including personal projects and "professional" work. Maybe it's because I wanted to implement all the subsystems myself, in that case not only do you have to learn database concepts but also caching, parsing, etc. I have mixed feelings about Rust and low level code, I really like Rust for high level code but building low level abstraction is still a pain. Anyway, good luck with your DB 🫡👨💻.
nice video
nice comment
📝
Taking notes? 👨💻
I like this kind of projects and not some management systems. creating version control system like git would be great project too.
Already planned, the problem with these projects is that they take a long time to build. I'm gonna do smaller projects from now on, because 7 months is too much 😂.
@tony_saro writing simplier key value store would be more easier as well as interesting especially when concurrent read writes are done. Implemenation datails doesn't really matter as far as you have such good explanations. Also little demos(although you had somwhere in the middle) at the end would be good motivation for staying tuned.
Only developing such project takes huge patience, dedication and time ofc. Not speaking about the creating, recording and uploading good quality videos here.
But anyways you did great job! Looking forward for more projects in future! Thanks
@tony_saro writing simplier key value store would be more easier as well as interesting especially when concurrent read writes are done. Implemenation datails doesn't really matter as far as you have such good explanations. Also little demos(although you had somwhere in the middle) at the end would be good motivation for staying tuned.
Only developing such project takes huge patience, dedication and time ofc. Not speaking about the creating, recording and uploading good quality videos here.
But anyways you did great job! Looking forward for more projects in future! Thanks
Yeah writing some key value in-memory database is definitely easier, I guess that's a project for another video. Thanks for the sub!
Hermano se cansó de los normies hispanos html css js y se cambió a la comunidad inglesa
😈😈😈
16th subscriber 12th like