Data Mesh Paradigm Shift in Data Platform Architecture
Vložit
- čas přidán 27. 08. 2024
- QCon San Francisco International Software Conference is back this November 18-22, 2024. Software leaders at early adopter companies will come together to share actionable insights to help you adopt the right technologies and practices.
Get exposed to new ideas and innovative approaches to software development and engineering, guaranteed to inspire and challenge you.
Don’t miss this opportunity to take your knowledge and skills to the next level and stay ahead in the fast-paced world of technology.
Register now: bit.ly/3Tc73rM
--------------------------------------------------------------------------------------------------------------------------------------
Video with transcript included: bit.ly/2TkpkDE
Zhamak Dehghani introduces Data Mesh, the next generation data platform, which shifts to a paradigm drawing from modern distributed architecture considering domains as the first class concern, applying platform thinking to create self-serve data infrastructure, and treating data as a product.
This presentation was recorded at QCon San Francisco 2019: bit.ly/38sivWf
The next QCon is QCon New York 2020 - June 15-17, 2020: bit.ly/2Uyj39C
For more awesome presentations on innovator and early adopter topics check InfoQ’s selection of talks from conferences worldwide bit.ly/2tm9loz
#DataMesh #Microservices #ParadigmShift
If are already a data guy skip to @18:30. Example at 38:06
God bless you
Excellent presentation and explanation for why a shift in how we manage data (and data architectural thinking) is needed. Powerful stuff. Thank you.
A really nice presentation from Zhamak Dehghani with lots of great points but no "Paradigm Shift" more like "Atomic Clock Drift"
She is declaring the crisis of Data without prove. Although its an interesting concept,
and I'm open to it, the Kuhn's paradigmen-shift for solving the crisis of data has a longer history.
Just recently the crises of data was solved by introducing the datalake,
before that the crisis of data was solved by the datawarehouse.
Will we, like in our beloved JScript "Ecosystem", get every year a new revolutionary framework?
Let's see . But give Data Mesh a try :-)
I think the key idea here is to have a domain/product owner team that owns an area end to end and their KPI is to keep their customers happy. We should not have an explosion of copies of data like in her approach. A data warehouse/lake is still sufficient, what is needed is a "meta" paradigm shift, not an architectural one.
Data mesh actually leads to far fewer copies of the data when done correctly.
@@datameshlearning How?
Thank you Ms Zhamak, Learnt a lot.
Excellent talk. This is a powerful paradigm shift
Thank you! Few interesting take away from this sessions.
Thiis boils down to two important things why you are doing what you are doing !
I am not advocating a pipeline first approach ever because that is the how and not the why!
How can be spelt out, tech stack can be defined. Scale, governance and other requirements can be addressed.
In my view, we cannot opt for a completely data decentralised domain orientation as well as we have seen historically with our journey with data Marts and looking at both top down and bottom up approach to data warehouses.
We need a common data model to address certain business requirements and have a single version of truth across the enterprise for specific business problems.
So a mix of current and proposed architecture pardigmn will prove relevant as we continue to evolve in our data journey and continue to innovate at scale.
100% agree Ruchi! This is a great blurb! Are you a practitioner, consultant, or other?
Have you looked into Data Vault 2.0 Methodology?
I liked it as a new voice into the debate but as far as what she has shown, Data Mesh is not a real need since most of the problems she highlights have not been a problem for quite some time - about a decade or more. The way she reports about BI/DW situation sounds more like she hasn't been in touch with the field for a couple of decades and do not know how things are really working today.
There are a LOT of failed projects around (Gartner ranges it into 70-80%) but most of those failures stem from bad choices and bad management, and not lack of tech options. Those problems are not solved by some word shuffling but rather with hard and steady work with people, not technology.
Just to make it clearer: DW/BI project problems are with people, not technology and have been so for at least more than a decade (well before Hadoop, for instance.)
excellent explanation
TLDR; skip to 37:19
this needs to be higher...
Thank you
Meh... this is no paradigm shift... it's the same abstract concept during mono -> microservices application dev transition.
She said inconvenient truth as well, as al gore, she just want to sell her stuff, exagerate it to the max
this is easier said than done
And it's not easily said. Zhamak isn't shying away from the fact this is super early and will take a ton of effort to implement it even moderately well but the benefits are great. And it is really only for large companies with many domains.
Inspiring talk.. Thanks!
I'd really like to know where is the improvement from Data Warehouses to Lakes, at most they are complementary! And the new trend is the Data Lakehouse approach, wich is actually quite promissing: "Big Data" technology applied with the proven concepts and methods of Data Warehousing. I'm not sure about the data mesh, to me it sounds too good to be true ...
For Data Excellence - that is, for Scale, Speed, Agility, Quality and Value - the legacy got to go. It's time for paradigm shift. Seems that Data Mesh is the one that can deliver what we need. Strong Buy.
Starts at Data Mesh starts @ 18:10
Aren't these the very basics ?
Love you
What is risk for "over-meshing"?
Is the data mesh not a siloed approach? Just saying
Would anyone have a comprehensive "digest" of the evolution of operational & OLTP architectures that evolved from monolithic to micro-services' oriented? The DevOps evolution timeline would be awesome aswell, I'd like to compare them with the evolution of analytics data oriented architectures.
Sounds like just another "microservice" but for data-lakes
I am confused :/
XMR to the moon
This is a great talk, some of this is very high level, and doesn't quite explain the technical stuff, but I imagine the very detailed technical stuff isn't the stuff she's the expert on. Just the same, I think I have a shallow grasp of what she's saying.
Lol almost 6 years ago I implemented this.
Either not adopted or adopted and thrown out and replaced by the next new thing in 5-6 years.