Pathfinder Analytics
Pathfinder Analytics
  • 38
  • 186 225
Microsoft Fabric End to End Data Project for Data Analysts and BI Engineers
Welcome to this comprehensive Microsoft Fabric data project using New York City taxi data. This video will guide you through an end-to-end data project, focusing on Data Warehousing, Data Factory, and Power BI.
We'll use TLC Trip record data, stored in a Fabric Data Lakehouse, and transform it through staging and presentation layers using Data Factory activities, Dataflows, and Stored Procedures.
Pre-requisites ✅
- Microsoft Fabric environment,
- Basic SQL, and data warehousing knowledge
Timestamps⌚
00:00:00 Introduction
00:04:32 Downloading the NYC Taxi Data Files
00:06:39 Data Lakehouse Files for Landing Data
00:12:19 Pipeline from Landing to Staging
00:42:16 Dataflow from Staging to Presentation
00:59:18 End to End Data Processing
01:03:33 Semantic Modelling and Power BI
01:07:44 Replacing the Dataflow with a Stored Procedure
Links and Resources 🔗
GitHub wiki: github.com/malvik01/Fabric-NYC-Taxi-Data-Project/wiki
NYC Trip Record Data: www.nyc.gov/site/tlc/about/tlc-trip-record-data.page
Check out my Udemy Course 📚
- Microsoft Fabric: The Ultimate Guide: www.udemy.com/course/microsoft-fabric-the-ultimate-guide/?referralCode=94FDA5B8134E63965E92
zhlédnutí: 29 530

Video

Effortless Querying with the Visual Query Editor (Microsoft Fabric Data Warehouse)
zhlédnutí 286Před 2 měsíci
This video demonstrates how to leverage the visual query editor in the Microsoft Fabric Data Warehouse for quick and efficient query writing. The visual query editor offers a no-code experience, allowing you to create queries effortlessly. 🔗 Links and Resources - Microsoft Documentation: learn.microsoft.com/en-us/fabric/data-warehouse/visual-query-editor
Microsoft Fabric Simplified - A Technical Overview (+ Platform Demo)
zhlédnutí 7KPřed 3 měsíci
Welcome to this in-depth technical overview of Microsoft Fabric! 🚀 In this video, I break down Microsoft Fabric's main components, architecture, and various experiences. You'll get a demo of the platform to help you understand what it offers and how it unifies workflows across data ingestion, processing, and visualization. 🔍 Key Topics Covered (Timestamps): 00:00 Introduction 00:52 What are the...
Databricks SQL: Group By All
zhlédnutí 330Před 3 měsíci
Simplify your SQL with Group By All. Want to become a Databricks Data Analyst and learn SQL on the Databricks SQL platform? Check out my Udemy course: www.udemy.com/course/databricks-sql-for-data-analysts/?referralCode=78C6FFDBE3A7474B9607
End to End Data Project with Microsoft Fabric - Data Engineering, Data Factory and Power BI
zhlédnutí 18KPřed 4 měsíci
🚀Join me in this tutorial as we build an end to end data engineering and reporting solution using Microsoft Fabric. This tutorial uses the USGS Earthquake API to build an end to end data solution focusing on the Data Engineering, Data Factory and Power BI experiences. ⌚ Timestamps 00:00 Introduction to the Project and Solution Overview 03:20 Creating a Fabric Workspace and Data Lakehouse 06:48 ...
QuickVisualize in Microsoft Fabric: Embed PowerBI Reports in Jupyter Notebooks
zhlédnutí 604Před 5 měsíci
This video shows you how you can render a Power BI report in seconds from a Spark DataFrame in Jupyter Notebook. 🔗 Links and Documentation: learn.microsoft.com/en-us/fabric/data-engineering/notebook-visualization#create-report-visuals-from-a-spark-dataframe learn.microsoft.com/en-us/power-bi/create-reports/service-interact-quick-report pypi.org/project/powerbiclient/ github.com/Microsoft/powerb...
Start a Microsoft Fabric Trial without using a work email address
zhlédnutí 4,8KPřed 6 měsíci
Want to get a trial of Microsoft Fabric but don’t have a company email address? In this short video I’ll show you how to bypass this requirement and set up a Fabric Trial WITHOUT needing to use a company email address. ⌚Timestamps: 00:00 Introduction 01:00 Azure Account 02:36 Creating an Azure User 04:05 Starting the Microsoft Fabric Trial 📚Check out my Udemy Course www.udemy.com/course/microso...
Query Snowflake Data on Databricks with Lakehouse Federation
zhlédnutí 3KPřed 6 měsíci
This video explains how to set up Lakehouse Federation to run federated queries on Snowflake data that is not managed by Azure Databricks. 🔗 Links and Documentation: Run Federation Queries on Snowflake: [learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/manage-privileges/privileges](learn.microsoft.com/en-us/azure/databricks/query-federation/snowflake) 💻 Check out my Data...
Data Access Control with Databricks Unity Catalog
zhlédnutí 3,9KPřed 6 měsíci
Welcome to this straightforward and practical guide on Privilege Management with Databricks Unity Catalog. This video explains how to control access to data and other objects in Unity Catalog. ⌚Timestamps: 00:00 Introduction 00:28 Unity Catalog Security Model 02:28 Assigning Privileges Demo 🔗 Links and Documentation: Privilege Types by Object: learn.microsoft.com/en-us/azure/databricks/data-gov...
Databricks Unity Catalog: Catalogs and Schemas
zhlédnutí 4,2KPřed 7 měsíci
Welcome to this straightforward and practical guide on Catalogs and Schemas in Databricks Unity Catalog. Catalog: The first layer of the object hierarchy, used to organize your data assets Schema: Also known as databases, schemas are the second layer of the object hierarchy and contain tables and views. ⌚Timestamps: 00:00 Introduction 00:30 Catalog Explorer UI 04:34 SQL Syntax 🔗 Links and Docum...
Databricks Unity Catalog: Storage Credentials and External Locations
zhlédnutí 8KPřed 7 měsíci
Welcome to this straightforward and practical guide on Unity Catalog Storage Credentials and External Locations. Unity Catalog introduces several new securable objects to grant privileges to data in cloud object storage. - A storage credential is a securable object representing an Azure managed identity or Microsoft Entra ID service principal. - Once a storage credential is created access to it...
Databricks Unity Catalog: A Technical Overview
zhlédnutí 26KPřed 7 měsíci
Welcome to the first video of my Databricks Unity Catalog Series! Unity Catalog brings a new layer of data management and security to your Databricks environment, and with this technical overview, you’ll learn about these capabilities. ⌚Timestamps: 00:00 Introduction 00:17 What is Unity Catalog? 00:30 Before vs After Unity Catalog 02:01 Key Features of Unity Catalog 02:39 Administrative Roles i...
Enabling Unity Catalog on Azure Databricks: A Step-by-Step Guide
zhlédnutí 17KPřed 7 měsíci
Enabling Unity Catalog on Azure Databricks: A Step-by-Step Guide
Understanding Databases, Warehouses, Lakes, and Lakehouses
zhlédnutí 206Před 7 měsíci
Understanding Databases, Warehouses, Lakes, and Lakehouses
Databricks Extension for VS Code: A Hands-On Tutorial
zhlédnutí 6KPřed 8 měsíci
Databricks Extension for VS Code: A Hands-On Tutorial
Real Time Streaming with Azure Databricks and Event Hubs
zhlédnutí 22KPřed 9 měsíci
Real Time Streaming with Azure Databricks and Event Hubs
Create Azure account and Portal Overview | Episode 8 (AZ-900 Azure Fundamentals)
zhlédnutí 124Před rokem
Create Azure account and Portal Overview | Episode 8 (AZ-900 Azure Fundamentals)
Security, governance and manageability benefits of the cloud | Episode 7 (AZ-900 Azure Fundamentals)
zhlédnutí 625Před rokem
Security, governance and manageability benefits of the cloud | Episode 7 (AZ-900 Azure Fundamentals)
Reliability and predictability benefits of the cloud | Episode 6 (AZ-900 Azure Fundamentals)
zhlédnutí 677Před rokem
Reliability and predictability benefits of the cloud | Episode 6 (AZ-900 Azure Fundamentals)
Scalability benefits of the cloud | Episode 5 (AZ-900 Azure Fundamentals)
zhlédnutí 113Před rokem
Scalability benefits of the cloud | Episode 5 (AZ-900 Azure Fundamentals)
High availability benefits of the cloud | Episode 4 (AZ-900 Azure Fundamentals)
zhlédnutí 95Před rokem
High availability benefits of the cloud | Episode 4 (AZ-900 Azure Fundamentals)
Public, private and hybrid cloud models | Episode 3 (AZ-900 Azure Fundamentals)
zhlédnutí 186Před rokem
Public, private and hybrid cloud models | Episode 3 (AZ-900 Azure Fundamentals)
IaaS, PaaS and Saas cloud service types | Episode 2 (AZ-900 Azure Fundamentals)
zhlédnutí 157Před rokem
IaaS, PaaS and Saas cloud service types | Episode 2 (AZ-900 Azure Fundamentals)
What is Cloud Computing? | Episode 1 (AZ-900 Azure Fundamentals)
zhlédnutí 345Před rokem
What is Cloud Computing? | Episode 1 (AZ-900 Azure Fundamentals)
5 Not So Common Tips For Pandas Data Frames
zhlédnutí 154Před 2 lety
5 Not So Common Tips For Pandas Data Frames
Introduction to Data Analysis and Visualization Libraries in Python | TUTORIAL
zhlédnutí 3,4KPřed 2 lety
Introduction to Data Analysis and Visualization Libraries in Python | TUTORIAL
Introduction to Sunburst Charts in Plotly Express (Python)
zhlédnutí 824Před 2 lety
Introduction to Sunburst Charts in Plotly Express (Python)
Introduction to Treemaps in Plotly Express (Python)
zhlédnutí 11KPřed 2 lety
Introduction to Treemaps in Plotly Express (Python)
What is SQL?
zhlédnutí 1,3KPřed 3 lety
What is SQL?
SQL Commands: Data Manipulation Language (DML)
zhlédnutí 198Před 3 lety
SQL Commands: Data Manipulation Language (DML)

Komentáře

  • @moeeljawad5361
    @moeeljawad5361 Před dnem

    Hello Can you delete a visual from the embedded report? i tried an i couldn't find a way. Or is there a way to customize what visuals to be included in the power bi report, from inside the notebook itself. Like for example, i would like to have a slicer on column A, ColumnB a table visual and a line chart visual, based on df? Thanks

  • @arunsundar3739
    @arunsundar3739 Před dnem

    the point on adding role 'account admin' to user through account portal helped to find the root cause, thank you for sharing :)

  • @leoh2502
    @leoh2502 Před dnem

    The last bit won't work if the cluster has "Azure Data Lake Storage credential passthrough" enabled.

  • @DevMehta0
    @DevMehta0 Před 4 dny

    'WithWatermark' is useful for late arriving data, right? We could close the window ASAP we cross the window limit but we are waiting for couple of minutes extra than our upper window limit.

    • @pathfinder-analytics
      @pathfinder-analytics Před 4 dny

      @@DevMehta0 that is correct, the window frame itself remains the same but we are extending the time to close to allow for any lag in data arrival

  • @DevMehta0
    @DevMehta0 Před 4 dny

    Loved the tutorial, Thank you Man 🙌

  • @zombiezombie1977
    @zombiezombie1977 Před 6 dny

    Working on this in Terraform and this video demonstrates steps very well. Great help, thanks :)

  • @LakshmiDv-m9n
    @LakshmiDv-m9n Před 7 dny

    manage account option is not available for my databricks workspace eventhough iam a global admin

  • @jessefields1550
    @jessefields1550 Před 12 dny

    This was very helpful. Thank you so much for posting this. I'm excited to try out Microsoft Fabric.

  • @Curious_human_
    @Curious_human_ Před 13 dny

    I followed this steps but i am unable to become fabric admin

  • @amynchauthani5122
    @amynchauthani5122 Před 18 dny

    Appreciate your clear and conscise intro on MS Fabric. Thank you

  • @imvikraman
    @imvikraman Před 19 dny

    So, I did manage to create an user and signed in to fabrics portal with it, and activated the Fabric Trial. But my fabric homepage has only Power BI and Synapse Data Science enabled. Help please ? :(

  • @amynchauthani5122
    @amynchauthani5122 Před 19 dny

    Excellent explanation with clear examples how to implement end to end data flow from source thru Pipeline to the PBI Visual. THANK YOU

  • @KurtJ-r8w
    @KurtJ-r8w Před 21 dnem

    Very nice. Thanks

  • @debasisbera5627
    @debasisbera5627 Před 23 dny

    Excellent. Highly appreciated

  • @ChristianoPiccinin
    @ChristianoPiccinin Před 26 dny

    great video brother

  • @Magmatic91
    @Magmatic91 Před 28 dny

    very good content, you earned a new subscriber. Could you please just explain the part where you set a filter for start date before appending to the gold layer, you say it's for avoiding duplication but I'm not sure I understand what you mean by this.

  • @araujoao
    @araujoao Před měsícem

    Thanks a lot for your video series. They are clear, well thought out, and meaningful. I appreciated and learned from the videos.

  • @KurtJ-r8w
    @KurtJ-r8w Před měsícem

    Great video. I am subscribing. Thank you. However, I am not seeing the tables when I go to SQL. Any suggestions?

    • @pathfinder-analytics
      @pathfinder-analytics Před 26 dny

      Hi, I think this is currently a bug with the schema enabled lakehouses. I understand that this is currently an open ticket.

  • @thedailyepochs338
    @thedailyepochs338 Před měsícem

    Hey, Nice video, is it possible to use the access connector and datalake automatically created with workspace or we leave it alone and why?

  • @insghtsandmore
    @insghtsandmore Před měsícem

    Just starting looking into Fabric, this tutorial gave me a clear overview what Fabric can do. Thanks alot!!

  • @hasanakhtar2120
    @hasanakhtar2120 Před měsícem

    Please create a Udemy course on Microsoft Fabric?

  • @elprofesornet8897
    @elprofesornet8897 Před měsícem

    Just great!

  • @Chris-ot1rf
    @Chris-ot1rf Před měsícem

    Literally no reason whatsoever that it should be that convoluted. Absolute classic haha. Thanks mate.

  • @Robertlavigne1
    @Robertlavigne1 Před měsícem

    Love these end to end tutorials! Excellent examples of what can be accomplished. Thanks for making these!!! For anyone attempting a similar workflow in production it’s likely a good idea to have the staging load dates be dependent on the most resent presentation load logging dates rather than the staging ones. With the setup shown in this video a failure between the staging and presentation loads will cause a month to be missing in the presentation layer that will require manual cleanup of the logging table to correct.

  • @mainuser98
    @mainuser98 Před měsícem

    the HR catalog, what path did u choose when you configured the managed location path?

  • @sreeramch
    @sreeramch Před měsícem

    Cant thank you enough...you explaination is crisp and clear. Thank you for providing free material which could hve cost few hundred dollars.

  • @piyug6810
    @piyug6810 Před měsícem

    How to learn microsoft fabric

  • @vchandm23
    @vchandm23 Před měsícem

    This is amazing, end-end show of how this complex stuff works. Is this the same if we already have a databricks workspace (premium tier) with no Unit Catalog? 1. we just have to create Access Connector Resource object 2. Do the same steps as recommmended here. Kindly confirm.

  • @nicemovies99
    @nicemovies99 Před měsícem

    Thanks mate, can't wait to go through this project!

  • @sharunkumar4806
    @sharunkumar4806 Před měsícem

    Thanks a lot Malvik, you are amazing!

  • @fidelixwashington
    @fidelixwashington Před měsícem

    Great content, congratulations! For future videos on Microsoft Fabric, it would be interesting to see a logical update process instead of append and also the formation of a star schema between layers.

  • @drisselfigha3547
    @drisselfigha3547 Před měsícem

    Excellente vidéo

  • @arun06530
    @arun06530 Před měsícem

    nice video

  • @usmanrahat2913
    @usmanrahat2913 Před měsícem

    How would we turn on the intellisense?

  • @Rider-jn6zh
    @Rider-jn6zh Před měsícem

    Can you please let me know licensing cost for fabric PowerBI for below users for users who will create the report using copilot. for users who will just do prompts and derive insights using copilot...

  • @Heart24945
    @Heart24945 Před měsícem

    Good .Thank you

  • @himanshuaggarwal91
    @himanshuaggarwal91 Před měsícem

    Beautifully described.

  • @Rider-jn6zh
    @Rider-jn6zh Před měsícem

    i dont see tenant setting in admin portal.. i see only 3 options in admin portal ----1.capacity settings 2.refresh summary 3.help+support

    • @pathfinder-analytics
      @pathfinder-analytics Před měsícem

      This probably means you’re not an admin in your Fabric workspace, if you’re using your organisational account you’ll need to speak to your team to assign you the relevant permissions

    • @trevoratnip
      @trevoratnip Před měsícem

      @@pathfinder-analytics I've utilized the free trial of Fabric based on your previous video- Thank you! I am still not able to access the map visuals setting, though. Do you know if it is possible on the Fabric free trial? Thanks.

    • @Curious_human_
      @Curious_human_ Před 13 dny

      same for me and I am also using free trail account

  • @neelshah1651
    @neelshah1651 Před měsícem

    Great video!!, Keep posting new one

  • @arshadnehal246
    @arshadnehal246 Před měsícem

    Very well explained. I was frustrated with only theory of the unity catalog

  • @krishnakumar-vi3tf
    @krishnakumar-vi3tf Před měsícem

    Does external table created in UC using parquet get auto refreshed when additional parquet files are added?

  • @bharatarse1503
    @bharatarse1503 Před měsícem

    Amazing I will definitely try this project in Microsoft Fabric step by step... Thanks for sharing very useful..😊😊..Keep sharing..🤟🤟.

  • @kvin007
    @kvin007 Před měsícem

    This is way better than official Databricks Academy courses. Thanks for sharing

  • @mahdinematshahi
    @mahdinematshahi Před měsícem

    Hi thanks for your excellent training video. I followed step by step but when I wanted to read json file in Bronze layer I faced this error" Since Spark 2.3, the queries from raw JSON/CSV files are disallowed when the referenced columns only include the internal corrupt record column (named _corrupt_record by default). For example: spark.read.schema(schema).csv(file).filter($"_corrupt_record".isNotNull).count() and spark.read.schema(schema).csv(file).select("_corrupt_record").show(). Instead, you can cache or save the parsed results and then send the same query. For example, val df = spark.read.schema(schema).csv(file).cache() and then df.filter($"_corrupt_record".isNotNull).count()." thanks for your help in advance

  • @barbarosyonar975
    @barbarosyonar975 Před měsícem

    Great content. Thanks a lot.

  • @avinkohale5853
    @avinkohale5853 Před 2 měsíci

    Amazing overview!😁 I could think of one drawback of unity here: say user1 has access to catalog1, workspace1 and workspace2 has access to catalog1 as well. That would mean that user1 now has access to workspace1 and workspace2. is that correct? If yes then is there anyway to prevent this?

    • @vidyakothawale1214
      @vidyakothawale1214 Před měsícem

      yes. according to me there is no option to prevent access for other workspaces for same catalog

  • @oscarestorach6162
    @oscarestorach6162 Před 2 měsíci

    Thanks a lot from sharing your knowledge

  • @user-re7tr3fh3x
    @user-re7tr3fh3x Před 2 měsíci

    But in my laptop when i try to create new connection i able to see databases of another tables

  • @mehdikefi2013
    @mehdikefi2013 Před 2 měsíci

    Insightful. Thanks a lot for the work and the sharing!

  • @egi4760
    @egi4760 Před 2 měsíci

    thanks for the demo, it was really helpfull