Introduction to Dataform in Google Cloud Platform

This tutorial shows the overview of the Dataform service in the Google Cloud Platform.
Link to the GitHub repo with workflow: github.com/rafaello9472/dataf...
Dataform documentation: cloud.google.com/dataform/doc...
00:00 Introduction
00:42 What is Dataform?
01:10 Key Features and Benefits
02:30 Why Dataform?
02:43 Key Concepts
04:19 From SQL to SQLX
04:57 Version Control and Collaboration
06:03 Dependency Management
07:30 Javascript in Dataform
09:50 Workflow Execution Scheduling Options
10:35 Creating Dataform Repository
11:41 Creating Necessary GitHub Assets
13:41 Adding Access Token to Secret Manager
14:18 Adding IAM Roles to Service Account
15:46 Creating Development Workspace
17:52 Intro to Demo Examples
18:36 Overview of Repository | 1
20:18 Running First Workflow & Adding IAM Role | 2
25:22 Dependencies as DAGs | 3
27:48 Custom Operations & Tags | 4
32:43 Assertions | 5
36:23 Skipping Pipeline Step Execution with Assertions | 6
38:55 Reuse JavaScript Variables in SQLX Files | 7

[@user-gh7bn1xj9d]

Thank you for this amazing video! It provides a great overview of the features and capabilities of dataform. I love how you started with key concepts and also included a demo at the end for a hands-on session. Personally, I've faced challenges using Jenkins for collaboration and lack of integration with version control systems. Dataform solves this problem. Finally, a really good presentation and I'll be checking out your channel for more videos in the future.

[@user-vl7ju6fh8g]

I am so excited about your examples
like Java print depository and all others you’ve mentioned

[@user-vl7ju6fh8g]

I am so excited about the whole explanations

[@user-vl7ju6fh8g]

Thanks again and again for sharing this Database Depository code

[@user-vl7ju6fh8g]

This is awesomely exciting news about your business account

[@saurabhbhardwaj3112]

Thanks for sharing this brief and informative tutorial! Really helpful👍

[@user-vl7ju6fh8g]

Thanks so much for your concern about this wonderful program on Database form for businesses online

[@Najla-us6pz]

Thank you so much for this great video 🌸

[@59600muslim]

Thank you ! very clear your explanations !

[@igorbulenko6335]

very useful, thanks for sharing

[@alperakbash]

Thank you so much for such a wonderful tutorial. Perfectly designed and structured. And also thank you so much to help me to meet such a powerful platform.

[@cloud4datascience772]

Thank you for the kind words!

[@an.laskevych]

This video was very helpful for me! Thank you 👍

[@DawidCzerwinski]

Thank you! Very helpful

[@joaomanoellins2219]

Thank you! Great video

[@benedictcoltman1983]

Superb video, thank you

[@cloud4datascience772]

Thanks! Glad you liked it :)

[@iamkeithfajardo]

Great video :)

[@user-vl7ju6fh8g]

I appreciate your interest in sharing this wonderful business opportunities online with your friends and family members who love sharing their business opportunities online with their businesses

[@user-vl7ju6fh8g]

This is great

[@kirilllosik7054]

Thanks a lot for sharing very valuable insights about using Dataform! Could you please share more practical examples of using incremental tables and assertions? Keep going!

[@cloud4datascience772]

Hey, thanks for your comment, I am glad that you've found this useful. I am not planning to cover anything additional regarding the Dataform for the time being. Best way will be to refer to the official documentation for the above-mentioned topics: cloud.google.com/dataform/docs/incremental-tables
cloud.google.com/dataform/docs/assertions

[@carlosernestoelizondoreyes6978]

Thank you!😁

[@cloud4datascience772]

Glad you enjoyed the tutorial! :)

[@MegaLobo000]

Thanks :)

[@user-gv6du9py2s]

Thanks for the great video. I was wondering if in feature-7, it's possible to use "docs.js" across repositories?? thanks

[@AbhijitKumar-uw8fd]

Thanks, very informative video. You create a git repository with public mode, where you able to connect the private git rep too?

[@user-um1fk1bl7g]

Is it possible to read a file directly from Google Cloud Storage using dataform?

[@user-nx9bu5wz9k]

This is a great intro video! Thanks for sharing!
QQ: I'm trying to figure out how to make commits pushed to remote/origin appear under different usernames, rather than the one the token belongs to. Does anybody know if this is possible?

[@alphaalpha4595]

best indian ever love u from morocco snor and chih and sk7k7 and hatim l7waa

[@user-vl7ju6fh8g]

It’s very important to listen to this lecture and following same to be able to make online business accounts realized

[@AnneCastro-Intern]

Thank you so much!!
I have a question though, it was said that to link a third-party remote repository to a Dataform repository, you need to first authenticate it. Any thoughts on this?

[@user-vl7ju6fh8g]

Depository platform is a very wide program to use in creating your own Database platform

[@Najla-us6pz]

Could you maybe explain how to set the workflow for dataform? Thank you again for this video! :)

[@cloud4datascience772]

For the time being I am not planning to do another video on Dataform unfortunately, I am preety sure though that if something wasn’t covered in video it should be covered in documentation. Good luck! ;)

[@Juno_CA]

thanks for sharing. Our team at Uber can not link Github Repo to Dataform Workspace, it always shows "Must be a valid SSH URL." for the remote Git Repo URL. Could you please help with it? Thanks!

[@cloud4datascience772]

Hey, I haven't come across such an issue when working with the Dataform. I would try to make sure that you are setting up everything according to the Google documentation if that doesn't help you should reach out to Google support directly as it would be some platform bug.

[@mallninja9805]

May God have mercy on your soul if you want to use a service account other than the default. And you'd think Google would have integration with their "Google Cloud Source" git clone - but nope. Google cloud is a disjointed mess.

[@Satenc0]

How to use bigquery tables to read, make some transformations and write to other tables, with dataform?

[@Satenc0]

Is it possible to use javascript parametrizable variables on your sqlx files for the queries?