Do you know why the output of your DLT pipeline didnt give Streaming Tables? I wrote a similar pipeline where I read CDC from a SQL source, write into bronze using Autoloader in DLT, into a streaming table, then write into silver in SCD2 in DLT as well. Both bronze and silver tables are generated as streaming tables... In your example, you get normal Delta tables. I might confusing concepts... Also, do you have a video covering streaming tables and materialized views? Thanks!
It depends how you specify the tables. You can specify them as streaming tables or just simple Live tables which are materialized views. Streaming tables are continuously updated while materialized views get updated every few minutes or when you manually update them. docs.databricks.com/en/delta-live-tables/index.html
@@AthanasiouApostolos so only streaming tables dépend on checkpoint locations then from what I gather. ATM my streaming tables are using scheduled DLT pipelines so are not continuous. Thanks for your replies btw! One of the best databricks content creators for sure. 🤗
Yes exactly checkpoints are for streaming tables. When you use scheduled pipelines is essentially like doing batch jobs despite the fact you are using streaming tables. You need to use continuous to get the actual benefits of streaming tables. That is of course if you have a good budget hahaha. And thank you, much appreciated.
how do we handle out of otder data?
Big Thank you , Cant give much but subscribed and Liked .
Great video, just as always!
Thank you, glad you find it helpful!!
Do you know why the output of your DLT pipeline didnt give Streaming Tables? I wrote a similar pipeline where I read CDC from a SQL source, write into bronze using Autoloader in DLT, into a streaming table, then write into silver in SCD2 in DLT as well. Both bronze and silver tables are generated as streaming tables... In your example, you get normal Delta tables. I might confusing concepts...
Also, do you have a video covering streaming tables and materialized views? Thanks!
It depends how you specify the tables. You can specify them as streaming tables or just simple Live tables which are materialized views. Streaming tables are continuously updated while materialized views get updated every few minutes or when you manually update them.
docs.databricks.com/en/delta-live-tables/index.html
@@AthanasiouApostolos so only streaming tables dépend on checkpoint locations then from what I gather. ATM my streaming tables are using scheduled DLT pipelines so are not continuous. Thanks for your replies btw! One of the best databricks content creators for sure. 🤗
Yes exactly checkpoints are for streaming tables. When you use scheduled pipelines is essentially like doing batch jobs despite the fact you are using streaming tables. You need to use continuous to get the actual benefits of streaming tables. That is of course if you have a good budget hahaha. And thank you, much appreciated.
Quick Question : If a record is dropped from Source table i.e hard delete how does apply_changes handle it .
How to track lineage with apply changes as in my case lineage gets breaked?