Change Data Capture (CDC) Explained (with examples)

Change Data Capture (CDC) is the process of recognizing when data has changed in source system so that a downstream system can take an action based on that change.
It’s a very good way to to move data from your transactional databases to your data warehouses or data lakes with minimal latency. It’s also a good way to setup a real time data pipeline where other processes, like stream processors, can listen for changes in data and take actions accordingly.
0:00 Intro
0:18 What is CDC?
0:53 Example
2:27 How does it work?
5:38 Use Cases?
7:08 Outro
#cdc #database #systemDesign
Visit me at: irtizahafiz.com?
Contact me at: [e-mail]

[@nadavge]

Thanks, you kept it simple and easy to understand!

[@swyxTV]

good topic choice and visuals! subscribed, keep it up

[@irtizahafiz]

Thank you so much! Hope you enjoy the future videos too. Let me know if you have any feedback.

[@lesterlino3316]

Great explanation, thanks!!

[@irtizahafiz]

Glad you enjoyed it!

[@Daily_rand_memes]

thank you for this video! really informative!

[@irtizahafiz]

Glad it was helpful!

[@muhammadkaiser3544]

Thank you! This was very helpful.

[@irtizahafiz]

Thank you! I will start posting again soon, so please let me know what type of content interests you the most.

[@dataisfun4964]

Beautiful, thanks.

[@irtizahafiz]

Thank you! I will start posting again soon, so please let me know what type of content interests you the most.

[@rajaramau6370]

nice explanation . Thank you :)

[@irtizahafiz]

You are welcome!

[@achamac-donald9229]

Great explanation

[@irtizahafiz]

Glad you think so!

[@andynelson2340]

nice explanation

[@irtizahafiz]

Thank you! Glad you found it helpful : )

[@amlord68]

where is the code example??

[@nguyenngothuong]

thank

[@souravpakhira]

how to detect change in database schema like rename of table name or adding new column?

[@irtizahafiz]

That's a good point. TBH, I am not 100% sure.
I believe, you might have to update the connector, and then refresh the existing data back into Kafka.

[@souravpakhira]

@@irtizahafiz nvm I have already found the solution and have implemented it

[@khushaltrivedi9829]

is it near to realtime? if you have master db as rds where write will happen and u would want search as Elastic search but we need to stream data real time will this be real time?

[@irtizahafiz]

Depends on "how" real time your application needs to be. If you are feeding the CDC data into ES, I believe you will need to re-index which will take time.
Personally, I haven't used that pipeline before, so I don't have too much context.

[@kartech4592]

Lets say I have a order booking system that has order and order details table. Now , one order details has changed. I want to send a complete order event that comprises of order and order details to kafka so that it can be consumed and stored in a time series database as a complete order model. Where exactly will the order details be fetched , because CDC will only tell me order details has changed.

[@irtizahafiz]

Hi! Thank you for asking.
So this is a very good use case of Kafka Streams. Let's say you have a CDC stream for `order` and one CDC stream for `order_details`. The two of these should be in two different kafka topics.
Using Kafka Stream or KSQL, you can join the two streams whenever either changes. Do the join based on the `order_id`. Check this out: supergloo.com/kafka-streams/kafka-streams-joins-examples/

[@kartech4592]

@@irtizahafiz Thank you so much for the explanation. I went through the kstreams join example.Lets say my kafka topics store only last 7 days worth of data. Now lets say 20 days later the order details changed so it was sent as an event to order details event topic. When I use kstreams to join with order, in the kafka topic to store order events, it wont find the order because its cleared out. So how is this handled in the above case?

[@mariofredrick1501]

how about upsert operation? is it supported by debezium?

[@irtizahafiz]

I believe it is.

[@frankdeng8]

How does the db send messages to kafka ?

[@irtizahafiz]

Hi! So there is usually a middle man between the DB and Kafka, something called a Connector. Debezium is a good example of that.
What the connector does is read from the database's log files and writes to Kafka. Most databases (if not all) has some kind of a log where it records every DB operation. You can replay all the changes by reading this file. For Postgres you have WAL (write ahead log) and for MySQL you have other bin logs.
So the connector reads from this log file and writes the changes to Kafka for every change you make to your data.

[@nr798yna]

Hi, its good explanation!. Could you make a video of how Microsoft SQL Server based CDC pushes messages to kafka ? I mean the implementation details! Thank You !

[@irtizahafiz]

Hi! I am not really familiar with Microsoft SQL Server, and currently its not in my plans :(

[@dendihndn]

is it safe to assume that CDC is just streaming concept of replicating & updating data between data sources?

[@irtizahafiz]

Yup! That's a really nice way to put it.

[@hp50537]

like I want to connect mysql to bigquery using pubsub how?

[@irtizahafiz]

There should be a Kafka connector you can utilize. I know Debezium has a few of them, but Google might also offer it as a service.
One option might be to use GCP's MySQL equivalent, if you want native integration with BigQuery.

[@joaopedrom6337]

Deus abençoe pelo tradutor automático do youtube

[@GabrielFerreira-is9ly]

Perfeito! gostaria de exemplos de uso em código como Node.js, Python

[@yossra-elhaddad00]

Thanks for this simple great explanation

[@irtizahafiz]

Glad it was helpful!

[@fniazi4u]

clickbait or whatever,, but waste of time.... where is the example of CDC???