Our DataMesh Journey: data architectures for microservices by Carlos Saona-Vázquez - Spring I/O 2022

Spring I/O 2022 - Barcelona, 26-27 May
What is the best data architecture in the age of microservices and “you-build-it-you-run-it” cross-functional, autonomous teams? How should different microservices share data between themselves to minimise coupling? How should microservices share data with offline data consumers such as BI, CRM, or ML training pipelines to minimise dependencies with the microservice teams? How can we reconcile data quality, trustworthiness and single-source-of-truth with autonomous teams releasing multiple times per day? Is this doable without a centralized data governance team? Are centralized data governance teams as proposed by data lake architectures compatible with autonomous teams and continuous delivery? Data meshes have been proposed as a new data architecture paradigm to answer the questions above, but there are very few public reports on actual implementations and their challenges. We will share the data mesh journey that we started 2 years ago in eDreams, one of the largest online travel agencies in the world. We will focus on three topics: motivation, technical solution and transformational approach. First, we explain why we think data meshes are a better fit for microservice architectures run by cross-functional, autonomous development teams. Second, we show the architecture of our current implementation, including how the mesh is connected to the online platform, the technologies we used, and the outcomes and rationales for each of the multiple trade-offs we made. They include decentralized team autonomy, data ownership, quality and accountability, PCI and strict (European) privacy regulations, self-service data access and reporting, infrastructure cost and budgeting. Third, we share the organizational challenges we faced when replacing the multiple data systems that were working well at the local level. Finally, we will conclude with our current list of open issues and future work.

[@songs4enjoy]

Quite an insightful & in-depth presentation that covered the most important problems we face in micro services world, solutions available & the trade offs amongst each of the solution chosen.

[@carlossaona5335]

The slides (PDF) can be downloaded from here: www.linkedin.com/posts/carlos-saona-vazquez_our-datamesh-journey-talk-2022-activity-6962322447907864576-YgU8