There are some of us that invested a lot in Databricks and want to stick with it due to its maturity, team, developer experience to name some. More than worrying about moving from Synapse which we never really took seriously anyway our train of thought is about how to integrate Databricks with OneLake ie use OneLake as storage layer and then connecting PBI to OneLake Lakehouse and no longer to Azure Databricks. Wonder if that makes sense...
Well should have read longer to find a likeminded person. My first thought was to use onelake exactly for this. Powerbi serving layer. Also if it's easy to use, business people could have permission to upload manual data directly to Onelake. Will be interesting times ahead 😁
It's a big move to drop dedicated SQL pools and move everything to delta. Very welcome. There will be some drawbacks, but minor in comparison to the mishmash of storage formats from before.
Can't help but be very critical about this whole new "tool". To me it feels like something Synapse was ought to become, but failed to become. Wasn't Synapse Studio supposed to become the universal Data & AI portal and ecosystem, integrating all the fancy tools like Data Factory, ADLS, Spark, ML Studio, and even Power BI? I hear a lot of "should be" and I miss everything Synapse lacked with things like mature CI/CD and good integration between components.
We should always remain critical, especially before building and operating an end-to-end data service. I was also complaining about data management, governance, and operations, as you properly pointed out. It seems they don't have yet that maturity to provide a real holistic framework beyond development. But I feel positive that they have at least embraced a single persistence format and that would ease many of these challenges. Maybe they will include all these other capabilities in an undefined future.....
Yep. They're careful to position this as an "evolution of Synapse" not a "we did a bad, let's try again". It's definitely closer to what the initial vision was, and the "Power BIness" of it makes it compelling to a lot of users. There's still a lot of growth and maturity needed as it goes through preview, as with anything!
@@rhambo5554Not at all. Its a more credible alternative for certain workloads, sure. It's a great dip into engineering for Power BI folk, absolutely. There will still be lots of use cases where Databricks is a more mature offering. We'll certainly still be working closely with DBX!
I caught you mentioning your channel at the Virtual Data+AI Summit - thanks for sharing! Eagerly looking forward to absorbing and learning from your content!
Where have I seen this diagram before? ... Ah yes Databricks lakehouse: cloud storage --> delta lake --> governance layer --> serving all usecases (BI, DE, Streaming, ML). I do not see any new ideas, this is even 2 years old story from Databricks. It is actually an acknowledgement from MSFT that Databricks took the right path from the beginning with the lakehouse approach.
True, but if being first meant anything we would all still be using Lotus 123. Microsoft has always been taking good ideas and then overtime dominating them.
Microsoft’s strategy, throw more broken products on top of other broken products. Would be nice if they just fixed what they have before they start adding additional layers.
Thanks you video is great. Great explanation and great energy. My only humble comment is to avoid having moving objects behind you, as it is a bit distracting. Apart from that, it is great. Subscribing now....
Yeah, I really need to actually sort out the scene frames, bring them up to date a little. Would like a little bit of movement, but not constant, eye-distracting movement! I'll hopefully get to it in the next couple of months! - Simon
Thanks Simon. Just the video I have been looking for because I was wondering the same thing about Synapse vs Fabric as I thought Synapse was going to be the 'One Stop Shop' for data analytics
Positives: Delta, Data Source Virtualization, Source Control Power BI Governance is massive, you really want an emphasis on curated data sets / models with appropriate data access control. Not a fan of the rate of re-branding by MS and having another iteration of the Azure analytics stack. Certainly need some clarity on data pipelines - Data Factory looks really watered down and where does Data Flows come in to play? Early days, would be good to see the public roadmap.
Can we use Databricks to create a lakehouse architecture and connect PowerBI directly to the Data Lake via Fabric’s “Direct Lake” feature? Or do we need to re-create delta tables in Fabric for PowerBI to access them?
You absolutely can use it like this. OneLake essentially becomes the data lake you associate with Databricks, then you can query the delta tables directly. The only thing it will miss is the "v-ordering" which gives a performance boost to how Power BI reads Delta
What about the industry DB, these industry specific template tables in synapse that we could model and use, with SQL language, did they move to Fabric?
Feel overwhelmed but am sure this is a step in the right direction. I imported a CSV file with a size of 117mb to OneLake and it compressed it to 24mb . Not sure how that compares to PBI/PQ but seems very impressive to me. What you do you think?
Yep, that's just good ol' fashioned columnstore compression at work! At the end of the day, you can use it quite simply like this - import data, let it do its thing, query it at speed through Power BI :)
Thanks for the breakdown video :) Synapse was not a good experience though. Version control was totally broken. Quality of life and organizing/tidiness of code is awful. Instead of fixing anything, I bet they changed the business model and some icons.
If F64 is compared to Premium P1 capacity within power BI which is available for 4995 USD but here in above chart i see the monthly charge of F64 to be 8409 , pls help me understand the diff . Is it only the power is same i.e. CPU and QPU ? but not the price
That's down to the annual commit discount. If you sign up for a P1 currently, you make a full 12 month commitment and the price is discounted accordingly. The price you're seeing for the F64 is the "monthly pay-as-you-go" price, which isn't discounted. There's a note in the capacities blog stating: "Additionally, later this year we will introduce an Azure Reservation commitment model for Fabric Capacities which will make their price comparable to the equivalent Power BI Premium capacities."
What a mess. Changing direction and strategy is an inherent part of MS culture. First (~2016) we had Data Lake gen1 and ADLA (we know how it ended). Then (~2018) we had heterogenous landscape of storage account based data lake gen2 + adf + databricks + azure data warehouse Then (~2020) we had an a great 'unification' promise that came with synapse And now we have this 'thing' It is very upsetting and frustrating being MS data ecosystem customer with these changes of directions and strategy all the time. If I invested millions of dollars building synapse based platfrom/dw (since MS marketing assured us many times this is their strategic bet and way forward), I'd be pissed right now. It is also frustrating being MS data platform professional. From strategy perspective Snowlake and Databricks seem much safer bets.
totally agreed. Giving the PowerBI has even worse reputation for version control and release management (except for upcoming update). And now they marketed as self-serviced data pipeline / ML. I think they'd better not look from business side of thing way too much.
It is eating this for lunch. I think fabric can maybe be okay if you are a small company and cant afford platform engineering and dont have too much data and too complex governance requirements. But fabric will most likely fail in an enterprise setting.
Thanks for the video, it's nice and clear introduction. The only point I didn't get is how to centrally manage metadata. Is there any central catalog? Or Fabric relies on Table format of Delta Lake? If so, is there a way to get list of all tables?
I think the most interesting part is 'LakeHouse' as a concept is new, they are encapsulating ADSL2, Delta format and SQL Serverless into one thing called 'Lakehouses' Moving away from Synapse workspaces to each LakeHouse with its own Serverless Endpoint. Not sure OneLake is 100% how you describe it, when creating a 'LakeHouse' it spins up ADLS2 on OneLake for each 'LakeHouse'. Looks like this could be multiple accounts & containers.
Don't worry, we'll do a deeper dive into OneLake, OneSecurity and all those pieces. Should clarify what's actually happening when you create workspaces, lakehouses etc!
I have seen a Private Preview version of this few months ago. Frankly, I am not impressed. Just repackaging everything exisitng into one platform. Nothing innovative or fundamentally unique. My main concern is not having control over the compute. What if our business is ready to write a fat check, but want the data pipeline to be faster? What levers do I have other than prayer? How can I commit to SLA's? How to cope up fast growing data volume and still meet my SLA? Also, coming from an Enterprise BI background, where is my semantic layer (Analysis service) and catalog? Power BI is good for simple dashboards, but not for large enterprise models with 100+ tables. The biggest positive is the Delta format as the standard. But since their spark engine is terrible, I would rather use Databricks SQL warehouse to consumption layer.
Yeah, the capacity model is fairly confusing at first, despite attempts to make it "One compute, just like Power BI". The story is that you can separate capacities out, so the business can pay for their reporting, use a separate bucket for engineering etc. We'll do a deeper dive vid in a week or two!
@Advancing Analytics aren't they getting rid of capacity though in favor of azure skus that are all pay as you go? You're thinking these separate skus use different compute?
I Hope Microsoft fixes the code diffs in Fabric. When I tried to diff Git changes in ASA, a horrid mess was displayed because ASA relied too much on JSON to store pipeline steps and notebook cells. In contrast, code diffs on Databricks notebooks and workflows are pretty clean to read because there isn't a ton of JSON goo cluttering things up.
Sooo that'll be where Microsoft Purview comes in, which sits outside of Fabric as it covers a wider remit than just analytics. Certainly for the glossary & catalog elements. Quality, I've not caught up on the roadmap yet.
So we were just getting quotes to setup Azure Synapse data lake environment the day of the announcement. Fabric is much more attractive. One of my big questions not yet answered is the cost model. What implications for the total cost of ownership? My headache just got worse trying to sort this out.
It's both easier and harder, all at once. There's a single cost model - just pick the size of capacity you need and you pay a single price. On the face of it, really straight forward. However, each workload you use consumes part of that capacity, and you need to level it out across what 24 hours of usage looks like... Which is hard. Should be doing a dive into capacity planning soon, but the general promise from Microsoft is that it will have a lower TCO... But that's yet to be proven out!
Really interesting direction 😊My first question with a all SaaS solution and integration of OneLake and Power BI. How do we work with test environments, deployment and releases? Usually we separate our storage per environment but only have one Power BI environment but deployment pipelines ensures it point to the correct storage. Do you only have one Environment/Capacity where you develop?
Quotation from Microsoft website - "Microsoft Fabric enables the novice user through to the seasoned professional to leverage Database, Analytics, Messaging, Data Integration and Business Intelligence " - do you think this is true?
I'm going to answer this myself, as I saw you can create shortcuts to your Databricks storage in OneLake and as such use the delta table directly in Power BI. I do assume you need a Premium license to do this.
Why did they chose to name this 'Fabric', while 'Data Fabric' means something very different? Just to mess with the millions of people who still haven't figured out what data mesh/ fabric means?? 😂
Repackaging, man. I've never seen MS as revolutionary but rather evolutionary. Either way they do a nice job of abstraction to the citizen developer and encourage role collaboration. That is MS' strength..... enabling role collaboration.
I was happy to see git integration finally, but looks like it's only for Fabric, thats not that great, I Can see that not all teams would like to use Fabric just now.
isn't this the reason why Databricks is present and solving most of the problems highlighted for Fabrics?
Exactly!
There are some of us that invested a lot in Databricks and want to stick with it due to its maturity, team, developer experience to name some. More than worrying about moving from Synapse which we never really took seriously anyway our train of thought is about how to integrate Databricks with OneLake ie use OneLake as storage layer and then connecting PBI to OneLake Lakehouse and no longer to Azure Databricks. Wonder if that makes sense...
Well should have read longer to find a likeminded person. My first thought was to use onelake exactly for this. Powerbi serving layer. Also if it's easy to use, business people could have permission to upload manual data directly to Onelake. Will be interesting times ahead 😁
If everything becomes Delta, isn’t it close to how Databricks work with current Microsoft product overhead?
It's a big move to drop dedicated SQL pools and move everything to delta. Very welcome. There will be some drawbacks, but minor in comparison to the mishmash of storage formats from before.
Definitely the best articulation I have watched so far!
Can't help but be very critical about this whole new "tool". To me it feels like something Synapse was ought to become, but failed to become. Wasn't Synapse Studio supposed to become the universal Data & AI portal and ecosystem, integrating all the fancy tools like Data Factory, ADLS, Spark, ML Studio, and even Power BI? I hear a lot of "should be" and I miss everything Synapse lacked with things like mature CI/CD and good integration between components.
We should always remain critical, especially before building and operating an end-to-end data service.
I was also complaining about data management, governance, and operations, as you properly pointed out. It seems they don't have yet that maturity to provide a real holistic framework beyond development. But I feel positive that they have at least embraced a single persistence format and that would ease many of these challenges. Maybe they will include all these other capabilities in an undefined future.....
Yep. They're careful to position this as an "evolution of Synapse" not a "we did a bad, let's try again". It's definitely closer to what the initial vision was, and the "Power BIness" of it makes it compelling to a lot of users. There's still a lot of growth and maturity needed as it goes through preview, as with anything!
@@AdvancingAnalytics Is this a Databricks killer?
@@rhambo5554Not at all. Its a more credible alternative for certain workloads, sure. It's a great dip into engineering for Power BI folk, absolutely. There will still be lots of use cases where Databricks is a more mature offering. We'll certainly still be working closely with DBX!
I agree with you. Fabric is already a failure for me. Databricks will keep eating their lunch.
You're extremely good at explaining and breaking things down. Thanks so much. I'm fully subscribed to your channel. Best elaboration ever!!!
best introduction to Fabric that I can find
I caught you mentioning your channel at the Virtual Data+AI Summit - thanks for sharing! Eagerly looking forward to absorbing and learning from your content!
Awesome content - thanks for detail and quick turnaround. Love the enthusiasm.
Excellent video as always, but am I the only one wondering where Azure DevOps fits? And what about unstructured data in the OneLake?
Where have I seen this diagram before? ... Ah yes Databricks lakehouse: cloud storage --> delta lake --> governance layer --> serving all usecases (BI, DE, Streaming, ML). I do not see any new ideas, this is even 2 years old story from Databricks. It is actually an acknowledgement from MSFT that Databricks took the right path from the beginning with the lakehouse approach.
True, but if being first meant anything we would all still be using Lotus 123. Microsoft has always been taking good ideas and then overtime dominating them.
Thank you for this! Great content.
One question prevails and it's who will be ready and feel confident enough to follow in the saga...
Does one lake do file versioning like one drive does?
Microsoft’s strategy, throw more broken products on top of other broken products.
Would be nice if they just fixed what they have before they start adding additional layers.
Excellent video, Simon!
Thanks you video is great. Great explanation and great energy. My only humble comment is to avoid having moving objects behind you, as it is a bit distracting. Apart from that, it is great. Subscribing now....
Yeah, I really need to actually sort out the scene frames, bring them up to date a little. Would like a little bit of movement, but not constant, eye-distracting movement! I'll hopefully get to it in the next couple of months! - Simon
Fabric is awesome. Interested to see how ADO gets included.
Thanks Simon. Just the video I have been looking for because I was wondering the same thing about Synapse vs Fabric as I thought Synapse was going to be the 'One Stop Shop' for data analytics
MS is banking on Power BI's success and bringing other tools beneath its umbrella.
Lets take a moment to appreciate all the others that will be confused by Microsoft fabric and Azure Service Fabric.
I was wondering how Power BI Desktop fits in all of this? Are we finally moving to a fully web-based Power BI? What do you think?
#PowerQueryEverything !!!
When we have a lakehouse, then why need a Warehouse? Both are using the same Delta Storage.
One of the first questions everyone asks! Video coming out on Thurs, going through this very question!
Positives: Delta, Data Source Virtualization, Source Control Power BI
Governance is massive, you really want an emphasis on curated data sets / models with appropriate data access control. Not a fan of the rate of re-branding by MS and having another iteration of the Azure analytics stack. Certainly need some clarity on data pipelines - Data Factory looks really watered down and where does Data Flows come in to play? Early days, would be good to see the public roadmap.
Great summary. Thanks
It did help Simon, thanks!
Can we use Databricks to create a lakehouse architecture and connect PowerBI directly to the Data Lake via Fabric’s “Direct Lake” feature? Or do we need to re-create delta tables in Fabric for PowerBI to access them?
You absolutely can use it like this. OneLake essentially becomes the data lake you associate with Databricks, then you can query the delta tables directly. The only thing it will miss is the "v-ordering" which gives a performance boost to how Power BI reads Delta
What about the industry DB, these industry specific template tables in synapse that we could model and use, with SQL language, did they move to Fabric?
Gooood question. I'm sure we'll see the Synapse industry templates in Fabric eventually, no sign of them yet though!
Feel overwhelmed but am sure this is a step in the right direction. I imported a CSV file with a size of 117mb to OneLake and it compressed it to 24mb . Not sure how that compares to PBI/PQ but seems very impressive to me. What you do you think?
Parquet for the win
Yep, that's just good ol' fashioned columnstore compression at work! At the end of the day, you can use it quite simply like this - import data, let it do its thing, query it at speed through Power BI :)
Thanks for the breakdown video :) Synapse was not a good experience though. Version control was totally broken. Quality of life and organizing/tidiness of code is awful.
Instead of fixing anything, I bet they changed the business model and some icons.
Version control and release management is Microsoft’s weakness, and they don’t care?
If F64 is compared to Premium P1 capacity within power BI which is available for 4995 USD but here in above chart i see the monthly charge of F64 to be 8409 , pls help me understand the diff .
Is it only the power is same i.e. CPU and QPU ? but not the price
That's down to the annual commit discount. If you sign up for a P1 currently, you make a full 12 month commitment and the price is discounted accordingly. The price you're seeing for the F64 is the "monthly pay-as-you-go" price, which isn't discounted.
There's a note in the capacities blog stating: "Additionally, later this year we will introduce an Azure Reservation commitment model for Fabric Capacities which will make their price comparable to the equivalent Power BI Premium capacities."
Plenty of changes, some of them particularly promising. My question is, what about data governance? Why did they left Purview behind?
Purview appears in the Admin Monitoring Workspace so not sure if that is a cut down version or fully featured
What a mess. Changing direction and strategy is an inherent part of MS culture.
First (~2016) we had Data Lake gen1 and ADLA (we know how it ended).
Then (~2018) we had heterogenous landscape of storage account based data lake gen2 + adf + databricks + azure data warehouse
Then (~2020) we had an a great 'unification' promise that came with synapse
And now we have this 'thing'
It is very upsetting and frustrating being MS data ecosystem customer with these changes of directions and strategy all the time. If I invested millions of dollars building synapse based platfrom/dw (since MS marketing assured us many times this is their strategic bet and way forward), I'd be pissed right now.
It is also frustrating being MS data platform professional.
From strategy perspective Snowlake and Databricks seem much safer bets.
totally agreed. Giving the PowerBI has even worse reputation for version control and release management (except for upcoming update).
And now they marketed as self-serviced data pipeline / ML. I think they'd better not look from business side of thing way too much.
I lot of these features under the hood are the same, it’s just a new wrapper and branding. Hopefully they have a migration tool.
Any thoughts on Azure Databricks’ place/role in this new ecosystem?
It is eating this for lunch. I think fabric can maybe be okay if you are a small company and cant afford platform engineering and dont have too much data and too complex governance requirements. But fabric will most likely fail in an enterprise setting.
I wonder if I need to connect a million private endpoints to get this on a private network.
Yes
@@jordanfox470 Thanks going to be sooo much fun.
I wonder about the networking aspect. For example, do i still need a gateway to go from pbi to fw-secured storage or to a no-public ip spark?
Thanks for the video, it's nice and clear introduction. The only point I didn't get is how to centrally manage metadata. Is there any central catalog? Or Fabric relies on Table format of Delta Lake? If so, is there a way to get list of all tables?
I think the most interesting part is 'LakeHouse' as a concept is new, they are encapsulating ADSL2, Delta format and SQL Serverless into one thing called 'Lakehouses'
Moving away from Synapse workspaces to each LakeHouse with its own Serverless Endpoint.
Not sure OneLake is 100% how you describe it, when creating a 'LakeHouse' it spins up ADLS2 on OneLake for each 'LakeHouse'.
Looks like this could be multiple accounts & containers.
Don't worry, we'll do a deeper dive into OneLake, OneSecurity and all those pieces. Should clarify what's actually happening when you create workspaces, lakehouses etc!
I have seen a Private Preview version of this few months ago. Frankly, I am not impressed. Just repackaging everything exisitng into one platform. Nothing innovative or fundamentally unique.
My main concern is not having control over the compute. What if our business is ready to write a fat check, but want the data pipeline to be faster? What levers do I have other than prayer? How can I commit to SLA's? How to cope up fast growing data volume and still meet my SLA?
Also, coming from an Enterprise BI background, where is my semantic layer (Analysis service) and catalog? Power BI is good for simple dashboards, but not for large enterprise models with 100+ tables.
The biggest positive is the Delta format as the standard. But since their spark engine is terrible, I would rather use Databricks SQL warehouse to consumption layer.
Yeah, the capacity model is fairly confusing at first, despite attempts to make it "One compute, just like Power BI". The story is that you can separate capacities out, so the business can pay for their reporting, use a separate bucket for engineering etc. We'll do a deeper dive vid in a week or two!
@Advancing Analytics aren't they getting rid of capacity though in favor of azure skus that are all pay as you go? You're thinking these separate skus use different compute?
Great explanation 👍👍👍I am wondering about source code and deployment as well as cost. Will wait to see your videos about them in next few weeks.
Read the data engineering part earlier. Seems like CI/CD and git integration are a
coming features. So we'll need to be patient with that
I Hope Microsoft fixes the code diffs in Fabric. When I tried to diff Git changes in ASA, a horrid mess was displayed because ASA relied too much on JSON to store pipeline steps and notebook cells. In contrast, code diffs on Databricks notebooks and workflows are pretty clean to read because there isn't a ton of JSON goo cluttering things up.
What about Data Governance layer? Business Glossary, Data Catalog and Data Quality?
Sooo that'll be where Microsoft Purview comes in, which sits outside of Fabric as it covers a wider remit than just analytics. Certainly for the glossary & catalog elements. Quality, I've not caught up on the roadmap yet.
So we were just getting quotes to setup Azure Synapse data lake environment the day of the announcement. Fabric is much more attractive. One of my big questions not yet answered is the cost model. What implications for the total cost of ownership? My headache just got worse trying to sort this out.
It's both easier and harder, all at once. There's a single cost model - just pick the size of capacity you need and you pay a single price. On the face of it, really straight forward.
However, each workload you use consumes part of that capacity, and you need to level it out across what 24 hours of usage looks like... Which is hard.
Should be doing a dive into capacity planning soon, but the general promise from Microsoft is that it will have a lower TCO... But that's yet to be proven out!
And obviously, get in touch with AA if you need another quote from a team already pretty deep with Fabric 😀
Really interesting direction 😊My first question with a all SaaS solution and integration of OneLake and Power BI. How do we work with test environments, deployment and releases? Usually we separate our storage per environment but only have one Power BI environment but deployment pipelines ensures it point to the correct storage. Do you only have one Environment/Capacity where you develop?
Fabric has workspaces like PBI
As usual Microsoft is pretty much clever to Market packaging.
So what was DataVerse ?
Quotation from Microsoft website - "Microsoft Fabric enables the novice user through to the seasoned professional to leverage Database, Analytics, Messaging, Data Integration and Business Intelligence " - do you think this is true?
I wonder if Power BI will also be able to use delta directly from i.e. Databricks like it will from OneLake
I'm going to answer this myself, as I saw you can create shortcuts to your Databricks storage in OneLake and as such use the delta table directly in Power BI. I do assume you need a Premium license to do this.
Purview and therefore Data Management is not a selling point for MS marketing. This makes me 😢
Why did they chose to name this 'Fabric', while 'Data Fabric' means something very different? Just to mess with the millions of people who still haven't figured out what data mesh/ fabric means?? 😂
Haha, who knows. Talk to Microsoft Marketing. I was not impressed when I heard the name, but it's not going to change now! 😅
@@AdvancingAnalytics Can we get a video on what this means for companies embarking on a data mesh journey (having not selected any tooling yet!)
Fabric is the dataBricks competition?
Too much of a bulk in one pocket. Interested to know how much it can hold with efficiency
Repackaging, man. I've never seen MS as revolutionary but rather evolutionary. Either way they do a nice job of abstraction to the citizen developer and encourage role collaboration. That is MS' strength..... enabling role collaboration.
Governance will be a headache
I heard that it should be integrated with purview. Hopefully it will be deep integration
Totally agree.
How much tho
Is this Microsoft's attempt to compete with Palantir's Foundry platform?
So when Fabric fails, do they finally just acquire Databricks?
Haha, that's a question for the Microsoft Ask Me Anything 😅. Let's hope it doesn't fail for now though eh? Competition in the space is good!
I'm not really buying this. Sounds like a bunch of mumbo jumbo to me. This is not clear, Microsoft. I have no idea what you're doing.
I was happy to see git integration finally, but looks like it's only for Fabric, thats not that great, I Can see that not all teams would like to use Fabric just now.
Not so i think