Azure Synapse Analytics
Azure Synapse Analytics
  • 248
  • 593 686
Native execution engine for Apache Spark in Fabric
Native execution engine for Apache Spark on Fabric is a vectorized engine that optimizes the performance and efficiency of your Spark queries by running them directly on your lakehouse infrastructure. The engine's seamless integration means it requires no code modifications and avoids vendor lock-in.
In this episode of Fabric Espresso, we delve into the product and technical details about the native execution engine. Estera Kot, along with Ankita Christine Victor from Fabric Engineering Product Group unpack the use cases, scenarios, and technical details about the innovation.
Blog: blog.fabric.microsoft.com/en-us/blog/public-preview-of-native-execution-engine-for-apache-spark-on-fabric-data-engineering-and-data-science?ft=All
Docs: learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview?tabs=sparksql
🎙 Meet the Speakers:
👤 Guest Expert: Ankita Christine Victor, Software Engineer at Microsoft.
LinkedIn: www.linkedin.com/in/ankita-victor/
👤 Host: Estera Kot, PhD, Principal Product Manager at Microsoft.
LinkedIn: www.linkedin.com/in/esterakot/
Twitter: estera_kot
👍 Liked this video? Don't forget to hit the 'Like' button and share it with your peers!
zhlédnutí: 2 253

Video

Fabric Copilot billing pricing
zhlédnutí 451Před 16 hodinami
We are pleased to announce the global availability of the public preview of Copilot in Microsoft Fabric. The preview includes enhancements for Power BI, Data Factory, and Data Science & Data Engineering. With Copilot now in preview, Microsoft Fabric offers a refined approach to data transformation, enrichment, and analysis, accelerating the journey to insights. In this installment of Fabric Esp...
Fabric Apache Spark Autotune and Run Series Job Analysis in Monitoring Hub
zhlédnutí 279Před 14 dny
In this episode of Fabric Espresso, we delve into two features: Apache Spark Autotune and Run Series Job Analysis. Estera Kot, along with guest Jenny Jiang, Principal Product Manager at Fabric Data Engineering, unpack the use cases, scenarios, and demos of these features. Autotune: learn.microsoft.com/en-us/fabric/data-engineering/autotune?tabs=sparksql Blog post: blog.fabric.microsoft.com/en-u...
Fabric Apache Spark Jobs monitoring capabilities - Resource Usage Analysis
zhlédnutí 330Před 21 dnem
Microsoft Fabric allows you to operate notebooks, jobs, and various applications using Apache Spark within your workspace. It also offers in-depth monitoring capabilities for analyzing your job execution statistics and logs. In this episode, Estera Kot joins Jenny Jiang, the Principal Product Manager at Fabric Data Engineering, to discuss Resource Usage Analysis. learn.microsoft.com/en-us/fabri...
Microsoft Fabric Data Engineering - Notebooks Monitoring and Apache Spark Jobs Advisor
zhlédnutí 353Před 28 dny
In this episode of Fabric Data Engineering, we delve into the world of Notebooks Monitoring and Spark Advisor. Our host, Estera Kot, along with guest expert Jenny Jiang, Principal Product Manager at Fabric Data Engineering, unpack the intricacies of monitoring Spark applications and advising on Spark. 🎙 Meet the Speakers: 👤 Guest Expert: Jenny Jiang, Principal Product Manager | Fabric Data Engi...
Microsoft Fabric Data Engineering - Apache Spark Jobs Monitoring
zhlédnutí 565Před měsícem
In this episode of Fabric Data Engineering, host Estera Kot and guest expert Jenny Jiang, Principal Product Manager at Fabric Data Engineering, dive into the world of Spark Monitoring. They explore the ins and outs of monitoring Spark applications, offering valuable insights and practical advice. 🎙 Meet the Speakers: 👤 Guest Expert: Jenny Jiang, Principal Product Manager | Fabric Data Engineeri...
Microsoft Fabric Data Science - Detecting Machine Failure (Classification).
zhlédnutí 379Před měsícem
Microsoft Fabric Data Science - Detecting Machine Failure (Classification).Predict when manufacturing equipment will fail. Explore the process of predicting machine failures with Fabric Data Science. This episode delves into classification methods used in detecting manufacturing equipment failures, providing valuable insights for both novice and experienced data scientists. 🎙 Meet the Speakers:...
Copilot in Microsoft Fabric | Ultimate demo of copilots in Notebooks - DS & DE
zhlédnutí 1,3KPřed měsícem
In this episode, we delve into the use of Copilot within Fabric Data Science and Data Engineering notebooks. Our host Estera Kot, along with guest expert Raj Rikhy, will showcase the different aspects of Copilot and how they can be effectively utilized within the notebook environment. 🎙 Meet the Speakers: 👤 Guest Expert: Raj Rikhy, Principal Product Manager | Fabric Data Science LinkedIn: www.l...
Optimize Warehouse costs with capacity pause and resume
zhlédnutí 690Před měsícem
Welcome back to another episode of Fabric Espresso DW series. In this episode, we'll be exploring capacity pause and resume features so you can minimize costs and maximize efficiency. 🎙 Meet the Speakers: 👤 Guest from Microsoft Fabric Product Group: Sowmya Sivaraman is a Senior Product Manager on the Microsoft Fabric (Data Warehouse) team, owning experiences like billing, pause/resume, cost mon...
Getting the most out of the Staging Mechanisms in Dataflows Gen2 Fabric Data Factory
zhlédnutí 858Před 2 měsíci
After you've cleaned and prepared your data with Dataflow Gen2, you want to land your data in a destination. You can do this using the data destination capabilities in Dataflow Gen2. With this capability, you can pick from different destinations, like Azure SQL, Fabric Lakehouse, and many more. Dataflow Gen2 then writes your data to the destination, and from there you can use your data for furt...
Automate your interactions with Fabric Warehouse
zhlédnutí 776Před 2 měsíci
Welcome back to another episode of Fabric Espresso DW series. In this episode, we'll be exploring how you can automate actions in your Fabric solution and streamline operations using API. 🎙 Meet the Speakers: 👤 Guest from Microsoft Fabric Product Group: Salil Kanade is a product manager on the Microsoft Fabric (Data Warehouse) team, owning developer experiences. LinkedIn: www.linkedin.com/in/sa...
Dataflow Gen2 data destinations in Fabric Data Factory
zhlédnutí 880Před 3 měsíci
Dataflow Gen2 is the new generation of dataflows. The new generation of dataflows resides alongside the Power BI Dataflow (Gen1) and brings new features and improved experiences. Join us in this Fabric Espresso episode as Miguel and Estera discuss Dataflow Gen2 data destinations (learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-data-destinations-and-managed-settings ). 🎙 Meet the Spe...
What are Dataflows Gen2 in Fabric Data Factory?
zhlédnutí 582Před 3 měsíci
Dataflow Gen2 is the new generation of dataflows. The new generation of dataflows resides alongside the Power BI Dataflow (Gen1) and brings new features and improved experiences. Join us in this Fabric Espresso episode as Miguel and Estera discuss Dataflows Gen2 and provide a comparison between Dataflow Gen1 and Dataflow Gen2. 🎙 Meet the Speakers: 👤 Guest Expert: Name: Miguel Escobar, currently...
Connect to new data sources from Power BI Report Builder
zhlédnutí 2,2KPřed 3 měsíci
Create paginated reports by connecting to 100 data sources with the familiar Get Data experience. Use the intuitive drag and drop power query editor to define M-queries that will be used to create the RDL dataset. 🎙 Meet the Speakers: 👤 Name: Nirupama Srinivasan, currently a Principal Product Manager at Microsoft LinkedIn: (6) Nirupama Srinivasan | LinkedIn Link to the video: learn-video.azuref...
Fabric Data Science Cust churn scenario - when a customer will stop doing business with the bank.
zhlédnutí 694Před 3 měsíci
The Synapse Data Science software as a service (SaaS) experience in Microsoft Fabric can help machine learning professionals build, deploy, and operationalize their machine learning models in a single analytics platform, while collaborating with other key roles. This article describes both the capabilities of the Synapse Data Science experience, and how machine learning models can address commo...
Fabric Data Science Sales Forecasting: Predict sales numbers for product categories at a superstore.
zhlédnutí 1,4KPřed 3 měsíci
Fabric Data Science Sales Forecasting: Predict sales numbers for product categories at a superstore.
Medallion Architecture Data Design and Lakehouse Patterns | Microsoft Fabric Data Factory
zhlédnutí 9KPřed 3 měsíci
Medallion Architecture Data Design and Lakehouse Patterns | Microsoft Fabric Data Factory
Microsoft Fabric Data Engineering - How to make the reference file work in Spark Job Definitions?
zhlédnutí 968Před 4 měsíci
Microsoft Fabric Data Engineering - How to make the reference file work in Spark Job Definitions?
Consume Warehouse data from other services
zhlédnutí 718Před 4 měsíci
Consume Warehouse data from other services
Fabric Spark Compute Capabilities - Azure VM's and their impact on performance
zhlédnutí 692Před 4 měsíci
Fabric Spark Compute Capabilities - Azure VM's and their impact on performance
Automated maintenance features in Fabric Warehouse
zhlédnutí 954Před 6 měsíci
Automated maintenance features in Fabric Warehouse
Getting Started with Microsoft Fabric
zhlédnutí 1,3KPřed 6 měsíci
Getting Started with Microsoft Fabric
CSV ingestion performance improvements
zhlédnutí 726Před 7 měsíci
CSV ingestion performance improvements
Autologging and MLFlow in Fabric Data Science
zhlédnutí 617Před 7 měsíci
Autologging and MLFlow in Fabric Data Science
Model & Experiment Tracking in Fabric Data Science (MLFLOW)
zhlédnutí 2,6KPřed 7 měsíci
Model & Experiment Tracking in Fabric Data Science (MLFLOW)
Microsoft Fabric Capacity Smoothing and Data Warehouse Throttling
zhlédnutí 3,1KPřed 7 měsíci
Microsoft Fabric Capacity Smoothing and Data Warehouse Throttling
Data Engineering Starter Kit - Quick Start with Fabric Product Group
zhlédnutí 1,2KPřed 7 měsíci
Data Engineering Starter Kit - Quick Start with Fabric Product Group
R (Programming Language) in Microsoft Fabric - An Insightful Overview with Fabric Product Group
zhlédnutí 1,1KPřed 7 měsíci
R (Programming Language) in Microsoft Fabric - An Insightful Overview with Fabric Product Group
Microsoft Fabric Spark Utilities - mssparkutils
zhlédnutí 1,7KPřed 8 měsíci
Microsoft Fabric Spark Utilities - mssparkutils
Permissions in Microsoft Fabric Workspace and SQL
zhlédnutí 2,5KPřed 8 měsíci
Permissions in Microsoft Fabric Workspace and SQL

Komentáře

  • @keen8five
    @keen8five Před 4 hodinami

    any chance you will bring this feature also to Synapse? 🙂

  • @jenilchristo8775
    @jenilchristo8775 Před 16 hodinami

    Does gluten works for spark scala dataframe apis?

  • @ManikumarAryas
    @ManikumarAryas Před 17 hodinami

    Great video and fantastic job! It's truly insightful.

  • @jayopachecoea
    @jayopachecoea Před 3 dny

    Me encantan los vídeos de este canal, cortos y precisos (y) !

  • @milad987
    @milad987 Před 4 dny

    Do the database templates in synapse analytics follow the star schema ?

  • @gauravchaturvedi3615

    how does data governance works with Domains defined in One Lake for different departments?

  • @lighteningrod36
    @lighteningrod36 Před 6 dny

    So, data sovereignty rules will limit the use if Co-Pilot in Australia, if my data is processed in the US.

    • @ruixinxu0130
      @ruixinxu0130 Před 6 dny

      Thank you for your feedback. Copilot in Fabric is powered by LLMs that are currently only deployed to US and EU. We are aware of the data sovereignty concern and actively expanding our deployment to more Geo regions. You can always check this link to get the latest info. learn.microsoft.com/en-us/fabric/get-started/copilot-fabric-overview#available-regions

  • @user-lj9fk8dg9h
    @user-lj9fk8dg9h Před 13 dny

    Hello sir, Thank you so much providing these productive videos. Today, I faced a challenge, and the solution I couldn't find elsewhere. That is How to Extract data from SAP Hana Cloud to Microsoft Fabric (cloud to cloud connectivity). Could you please help me here?

  • @user-ph1km5vk9l
    @user-ph1km5vk9l Před 13 dny

    whta is name of the episode mentioned with short cuts explained ? thank you

  • @keen8five
    @keen8five Před 21 dnem

    status "running" just says that the vCore "did something", right? But there is no way to tell if the cores were running at 1% or at 100% load, correct?

    • @jennyjiang6301
      @jennyjiang6301 Před 21 dnem

      Yes, you are right. The resource utilization chart currently only indicates that the vCore is running and does not indicate CPU or memory utilization. What kind of information are you specifically looking for?

  • @RSCHAB
    @RSCHAB Před 22 dny

    Hi How to add a table into the lakehouse? I dont have one.. br. R.

  • @ashanw
    @ashanw Před 22 dny

    Great explanation and good content. Can you kindly share the yml file with me? Thanks

  • @naimuddinsiddiqui9249

    Great explanation , If we will do any changes in dataset which is in our local pc how would it reflect the data changes in kql do we have to establish any bridge like Integration run time/Virtual/Cloud gateway ?

  • @olegkazanskyi9752
    @olegkazanskyi9752 Před 25 dny

    I get this error when I'm trying to clone a table. Any hints on how to resolve it? Feature 'DISCOVERED TABLE' is not supported by table clone.

  • @vishwanathvt7701
    @vishwanathvt7701 Před 25 dny

    I have created the Synapse workspace what is the username and password? How to set that?

  • @MrLee1334
    @MrLee1334 Před 27 dny

    Hey while working with parquet files ive noticed depending on sql query complexity it may occur that running the exact same SQL query multiple times for the exact same parquet file, it may result in different results - has anyone ever noticed that same behavior before?

  • @juanm555
    @juanm555 Před 28 dny

    Excellent video, Abhishek explains everything in a wonderful way. Eagerly expecting more videos with him!

  • @BUY_YOUTUB_VIEWS_378
    @BUY_YOUTUB_VIEWS_378 Před 28 dny

    🎉

  • @moeeljawad5361
    @moeeljawad5361 Před 29 dny

    Thanks for this video, i am currently using the notebook activity in Fabric Pipelines. My notebook is mature now and it runs very well. I was thinking of gathering the code in the notebook into a job definition, for the sake of saving execution time in the notebook. Would replacing a notebook by a job definition makes the code execution faster? Another question would be about job descriptions themselves, if you have defined some helper functions in the notebook, can i move them to a side job definition that is being called from the main job definition? if yes then how? Thanks

  • @keen8five
    @keen8five Před měsícem

    I'd love to see the Capacity Unit consumption of a Notebook execution in the Monitoring Hub

  • @rankena
    @rankena Před měsícem

    Is there a way to generate Bearer token programmatically?

  • @SumitArora-zf3of
    @SumitArora-zf3of Před měsícem

    What are the options to build a Power BI report on a large dataset if it contains lets say 500millions of records?

  • @user-dy8xu7uj8k
    @user-dy8xu7uj8k Před měsícem

    Hi, Good Morning!, I have to convert the existing SQL server stored procedure into fabric environment, In my stored procedures there are CURSOR commands but fabric doesnt support CURSOR commands, in this case how do I proceede, is there any alternative.

  • @Get_YT_Views_579
    @Get_YT_Views_579 Před měsícem

    Thanks for the positivity!

  • @peterlapic6761
    @peterlapic6761 Před měsícem

    Is there a way to perform Lifecycle Management policy on the Dataverse using Synapse Link? Want to pull all data from Dataverse to the Datalake the way Synapse Link does but delete old data in the Dataverse but still retain it in the Data lake. I want the data in the data lake to run through the Azure Lifecycle management policy so that it ends up in the cooler tiers to save cost but still be reportable for PowerBi using serverless sql.

  • @thepakcolapcar
    @thepakcolapcar Před měsícem

    hello @amit When I follow the steps, in power BI I see the "Trial" option as disabled adn by default it has selected my "Pro" licence. However on top it shows me "PPU trial: 59 days left". Is that how it is uspposed to be? Further as I proceed, and try to create Lakehouse, it gives me a message asking to upgrade to free Fabric tiral capacity.

  • @mcquiggd
    @mcquiggd Před měsícem

    Unfortunately, the audio is very bad, and also the screen resolution of these recordings makes it very difficult to read - the occasional zooming in just makes it confusing. It's a pity as the content is pretty good - perhaps include the example files so people can try this themselves. This series could really use an Editor to make sure the content is uniformly presented.

  • @adatalearner
    @adatalearner Před měsícem

    May I make a request to include a session on how an enterprise Fabric environment should look like including DevOps CI/CD pipelines ?

  • @adatalearner
    @adatalearner Před měsícem

    does this require separate co-pilot license ?

  • @i.k.986
    @i.k.986 Před měsícem

    maybe there are questions: what does the burndown do? The smoothing takes place always, I mean at least for specific activities, right? When the capacity is turned off, and activities are smoothed, during the off period of the capacity, the capacity is still charged, right?

  • @omerturkoglu4259
    @omerturkoglu4259 Před měsícem

    can we say that delta is based on parquet? So Delta is nothing but an advanced Parquet?

  • @adilmajeed8439
    @adilmajeed8439 Před měsícem

    Thanks for sharing. Why the copilot is not incorporating SynaspeML code instead of scitki-learn library? Once the data volume becomes large, scit-kit learn library will not work efficiently the way it needs to be, at the end the DataFrame is based on pandas not Apache DataFrame. Any suggestions on that?

    • @rajrik
      @rajrik Před měsícem

      Excellent question. We're working on improving our integration and native awareness of Fabric capable libraries such as SynapseML and you will continue to see those improvements emerge as Copilot progresses this year. Watch this space!

  • @EsteraKot
    @EsteraKot Před měsícem

    Clarification: we were planning to switch to the Microsoft Fabric www.youtube.com/@MicrosoftFabric channel, but we have finally decided to stay here. We will continue delivering more content for our nearly 13k loyal viewers. Thank you!

  • @gpltaylor
    @gpltaylor Před měsícem

    short and to the point! nice.

  • @gpltaylor
    @gpltaylor Před měsícem

    I like this style of demo breakdown where we're not treated like morons :) We can all read the microsoft learn website. Here we dig into each section. From this along I feel I am able to get work done. Thank you

  • @gauravdevgan79
    @gauravdevgan79 Před měsícem

    does it provide comparable features as offered by shiny app ?

  • @vijaybodkhe8379
    @vijaybodkhe8379 Před měsícem

    Thanks for sharing

  • @04mdsimps
    @04mdsimps Před měsícem

    I tried fabric last summer when it came out and deemed it a beta at that point. Now its a year on, why should I move from azure synapse and power bi to fabric?

  • @bitips
    @bitips Před měsícem

    Question : If I'm playing separately for storage, why my storage becomes inaccessible when my capacity is paused ?

    • @up_0078
      @up_0078 Před měsícem

      Data is accessible as long as you have compute available. You create databricks cluster or any other compute and can access data stored on onelake.

  • @Milhouse77BS
    @Milhouse77BS Před měsícem

    Looking forward to this.

  • @mattstainsby4542
    @mattstainsby4542 Před měsícem

    I want this to work so badly but the config is turning out to be really difficult. For my kernal I can see fabric-synapse-runtime, however, when I'm running, spark is not being recognised

  • @yashub9580
    @yashub9580 Před měsícem

    i am running a sarima but whenever i am running it its giving me 100+ experiments. But i should be getting only one

  • @i.k.986
    @i.k.986 Před 2 měsíci

    Thank you for this clear explanations!

  • @sabarivel4555
    @sabarivel4555 Před 2 měsíci

    How to reuse the common dimensions across semantic models like the one lake shortcut created in this demo? This would be really useful to reuse the shared dimensions across subject areas.

  • @knuckleheadmcspazatron4939
    @knuckleheadmcspazatron4939 Před 2 měsíci

    This is really awesome! For some files this is a great method. Use it when it works kinda thing.

  • @sanishthomas2858
    @sanishthomas2858 Před 2 měsíci

    Nice. if I save the files from source into the Lakehouse File as csv and Json then will it save it has delta parquet if not then why we are saying data is saved in one lake as delta parquet

  • @sam910312
    @sam910312 Před 2 měsíci

    This could blocked D365 transactions?

  • @bloom6874
    @bloom6874 Před 2 měsíci

    great series

  • @MrSatishc84
    @MrSatishc84 Před 2 měsíci

    I need the video for synapse deployment using YAML file

  • @bloom6874
    @bloom6874 Před 2 měsíci

    Learning is really quick with your videos. Why you stopped posting more videos in this series. Please continue.