Video

fatal: not a git repository (or any of the parent directories): .git I am getting this error, why not mention " if you are getting errors, then do this..." ?!!!!!! you should have thought about it !

Thanks for this video! Generally agree with all the points you mention, but what are your thoughts on allowing stakeholders to self-serve versus implementing controls such that most, if not all, data reporting should always be through the data team?

OMG! No Cheap chat?! A very straight forward tutorial!

So whats the difference between Inkon data warehouse vs traditional relational db?

Very good and easy explanation. GJ

Great content! I had a question - why would companies choose to use standalone ELT / ETL providers (e.g. Stitch, Matillion) over the native Amazon Glue / Azure data factory? Wouldn’t it be easier to use the cloud provides as it would be more integrated?

Totally agree! At my job we create all the business logic in the Gold Layer, there we can have it commented, keep track of changes, its way easier to debug problems, I've been the whole week checking some columns and updating them with the business team based on the way they want it changed. Data analysts just need to create reports and understand the data to present it to stakeholders/other teams, not that this is an easy task since they are essentially our bridge with the business team

This would work as a podcast instead of a video. No visual not examples

If I have a sql server database and I want to connect to it in Metabase and create a dashboard, and I want the dashboard to be live (the visuals refresh automatically when new data comes in the database), similar to Direct Query in Power BI, is that possible in Metabase?

Hey Kahan! There were some changes to get the dbt env setup going.. perhaps it's time for an update? Had to find work arounds.. when I do it, profiles.yml file doesn't exist or cannot be found.. anyways, thanks for making the tutorial

thanks!

What about the ingestion part? I mean, in which environment do you develop and test the ingestion logic? Or is this just for production?

The staging layer would be equivalent to a landing zone?

Okay, I'm not crazy. The audio IS OUT OF SYNC for this particular video and it's making it nearly impossible to watch this video.

How to avoid hardcoding of username and password?

That's pretty cool. You could theoretically capture the output of a process on Linux on one job, then feed that output to another job that runs-on Windows.

Nice! I use a staging layer as a view and then 4 more layers for the pipeline until I get to the mart. I usually alternate between views and materialized tables, but I am not quite sure how to know the optimal way to decide between tables and views at each time. How do you compare performance, storage and other practical factors?

why can't we Upload a txt file😢 i cant upload my Obfuscated script I don't want it being stolen😢

Can you do a follow up on how you might put some of these metrics into a database? I was under the impression that a semantic layer doesnt live in the traditional database and id love your guidance on where you might put it?

The only things that can't be pushed back to the database are the dynamic calculations like ratios that need to be recalculated in real time depending on the user's interaction with the report

actually no more START FREE and DEPLOY OPENSOURCE buttons there

Interestingly, user are always excited to do it themselves and later on outsource to the IT/Dev to do the change because of getting loss into the report transformation. 😂 and in database give you track of change with code repository.

Great insights & very informative ! I have a question : so in my dbt project, i got the staging folder populated with functional models (transformations and aggregations in fuction of source data in the yml file) and modelized with entity relationship diagram, so what is the utility of marts layer essentially ? Can I just add constraints/relationships… on this folder for specific models ?

Thanks a lot Bro 👊🏻 really appreciate ❤

►► Establish a Well-Structured Data Warehouse for Your Small Team In 90 Days (Free Guide) -> www.kahandatasolutions.com/guide

no configuration file found while cloning from github

Do you truncate the data each batch pipeline run on staging and capture the cleaned data in snapshots? If not, how do you avoid duplicates down stream if you’re using say SCD Type 2?

fking love you dude, been racking my brains on this for the last couple of hours!!

Brilliant

Please make videos on “Meltano” and how to use dbt within this? 🎉

Can you provide one video on snowflake azure oath integration for API access

Hi , How we can set DBT_VALID_TO default Null values to something other max number

Nice! Question: Do you also use views in your warehouse and mart layers? I've been at companies where the marts were basically views based on views based on views times 10 which was terrible for the performance of getting the data.

Nice work! Where should the staging layer come when using a bronze, silver, gold medallion structure ?

Stage All the Things

►► Establish a Well-Structured Data Warehouse for Your Small Team In 90 Days (Free Guide) → www.kahandatasolutions.com/guide

i don't understand why dbt created a view and not a table?

watching this videos as a BE dev, what a waste of my time. uselessness waste of my time

I just discovered your videos. They are excellent. Clear, concise and to the point. Great content! Thanks so much!

Thank you so much for this video! Really helpful!

Thinking of pet projects, I've got a lot of workout data in google sheets i'd like to automate loading into a database (I've manually inserted data into db before so trying to broaden more than that). What would you use/learn to automate loading data. Does airbyte -> dbt -> postgress make sense?

so there are absolutely no industry standards what so ever, and we should just make up our own internal rules. I agree having internal rules is the most important, but teams aren't closed ecosystems, it's preferable to talk about industry standards. action words should be all capitalized. most of these have preferred implementations.

The content is fantastic, but your editing needs work. It's really hard to listen and watch when the video looks and sounds so glitchy.