How to build a Durable Write Ahead Log in Go | Segmentation, Auto Recovery, Checksums
Vložit
- čas přidán 5. 07. 2024
- Link to the newsletter post: jyotinder.substack.com/p/unde...
Link to the complete code: github.com/JyotinderSingh/go-wal
Subscribe to my substack: jyotinder.substack.com
In this video we take a deep dive into understanding how Write Ahead Logs ensure durability in distributed systems, databases, and many such applications. We also take a deep dive into how WALs support high write throughput, data integrity, data recovery, and segmentation.
GitHub: github.com/JyotinderSingh
Website: jyotindersingh.com/
Timestamps:
00:00 - Introduction to Write Ahead Logs
0051 - How databases use WALs
01:10 - Structure of a WAL
02:09 - How WALs achieve high write throughput
03:15 - Segmented Logs
04:14 - WAL Repairs
05:16 - Implementation in Go
06:56 - Initializing a WAL instance
09:37 - Writing entries to the WAL
12:48 - Reading all entries
14:35 - Reading entries from offset
15:15 - Repairing corrupted segments
17:52 - Testing the implementation - Věda a technologie
nice explanation
Great video on a topic I was curious about. Thanks!
Great video please make more videos on low level stuff
What a great video! thanks for the clear explanation!
Thank you!
Great stuff! Thanks a lot! Please keep posting such videos with such clear explanation. 👌
Thanks! More videos and articles coming!
A very insightful video. Just had one question around repair. Once we find a corrupted log wouldn't the data be insconsistent even after the repair? I am coming from the place where I have to use repair for a DB recovery
The repair is essentially discarding all data that it cannot read. It preserves entries up till the last row of data that the system is able to read successfully. We work on the assumption that all data which we can read is correctly stored in the WAL (Reinforced by the checksum).
The recovery mechanism is on best effort basis, and attempts to retrieve all data that it can successfully read.