Criteo DevXDays - BigDataFlow: Continuous Delivery of data pipelines

This talk was presented in our Criteo DevXDays 2022 edition.
Data at Criteo is a core asset and the source of reports we provide to both audiences, external and internal. We are talking about a massive amount of data daily and need a proper workflow management system to orchestrate every piece involved.
After trying and experimenting with different solutions, an internal project started to create a better system that would fit our needs. The BigDataFlow project was born.
As a platform, BigDataFlow handles releases for users. To make continuous deployment possible without causing incidents, BigDataFlow has some useful tools: Static Analysis, and a Command Line Interface.
The following talk covers those two previous points, showcasing the safe experience users have when editing data pipelines.
You can also check the article here: / bigdataflow-continuous...
And, of course, feel free to share your insights in the comments!