![Anjan GCP Data Engineering](/img/default-banner.jpg)
- 87
- 347 950
Anjan GCP Data Engineering
India
Registrace 30. 04. 2022
This channel is created to serve below purposes
- Educate about Data Engineering concepts and related products available on Google Cloud Platform
- Content of the videos are more focused on practical aspects of the concepts and use-cases
- Each video will have high level discussion of concepts in theory and followed by respective hands on/demos immediately
- Content of the videos are made such a way to help beginners and professionals
- Series of product wise videos are uploaded based on complexity of the concepts ( Ex: Big Query, Dataflow, Dataproc..etc)
- Please make use of playlists created based on products.
- Educate about Data Engineering concepts and related products available on Google Cloud Platform
- Content of the videos are more focused on practical aspects of the concepts and use-cases
- Each video will have high level discussion of concepts in theory and followed by respective hands on/demos immediately
- Content of the videos are made such a way to help beginners and professionals
- Series of product wise videos are uploaded based on complexity of the concepts ( Ex: Big Query, Dataflow, Dataproc..etc)
- Please make use of playlists created based on products.
Write Big Query SQL queries with Gemini assistance
Write Big Query SQL queries with Gemini assistance
zhlédnutí: 132
Video
Basics of Service Accounts and IAM Roles
zhlédnutí 1,4KPřed 9 měsíci
Basics of Service Accounts and IAM Roles
Learn code free Data pipelines with Cloud Data Fusion | Packed with detailed Demos and explanations
zhlédnutí 7KPřed 11 měsíci
Learn code free Data pipelines with Cloud Data Fusion | Packed with detailed Demos and explanations
Serverless Change Data Capture (CDC) , Replication , Data Synchronisation with Cloud Datastream
zhlédnutí 1,8KPřed 11 měsíci
Serverless Change Data Capture (CDC) , Replication , Data Synchronisation with Cloud Datastream
ETL nested AVRO files into Cloud SQL, GCS using Dataflow Pipeline
zhlédnutí 1,1KPřed 11 měsíci
Code : github.com/anjangcp/GCP-Data-Engineering-Demo-Codes/blob/9b8b7b5c07129188237965b266f6d73f0350d5a0/Dataflow/batch_etl_avro_data_cloudsql.py Must watch below videos before you watch this : czcams.com/video/hWpNyGj5xVk/video.htmlsi=is1OHok0P0XMDRsS czcams.com/video/dKGB3d2XXiE/video.htmlsi=CvdRfpAeyl8NYaDd
Automate GCP Project’s IAM Roles Snapshot process using Cloud Functions, Bigquery, Cloud Scheduler
zhlédnutí 481Před rokem
Ref Code: github.com/anjangcp/GCP-Data-Engineering-Demo-Codes/blob/698552fb8352876303e3d41a2e3ea56ae552d103/Common_Realtime_Usecases/iam_snapshots.py
GCP Professional Data Engineer, re certified ,Exam experience and my suggestions to Aspirants
zhlédnutí 821Před rokem
GCP Professional Data Engineer, re certified ,Exam experience and my suggestions to Aspirants
Selecting GCP ETL/Data processing services | GCP Professional Data Engineer
zhlédnutí 876Před rokem
github.com/anjangcp/GCP-Data-Engineering-Demo-Codes/blob/3fe03fe8304dd34f3cfb3ae9ab43f47acf44b180/GCP_Data_Eng_concept_files/gcp_etl_services.png
GCP Professional Data Engineer | Storage and Database selection process, Test your knowledge
zhlédnutí 512Před rokem
GCP Professional Data Engineer | Storage and Database selection process, Test your knowledge
GCP Professional Data Engineer | Selecting suitable storage and database services based on use case
zhlédnutí 1KPřed rokem
GCP Professional Data Engineer | Selecting suitable storage and database services based on use case
Cloud Big Table - Introduction | Console , CBT - Basic Demo with Examples
zhlédnutí 1,7KPřed rokem
Cloud Big Table - Introduction | Console , CBT - Basic Demo with Examples
Import Bulk data from GCS into Spanner Table using Dataflow Jobs
zhlédnutí 890Před rokem
Import Bulk data from GCS into Spanner Table using Dataflow Jobs
Basic Cloud Spanner database operations using Python Client Libraries
zhlédnutí 580Před rokem
Basic Cloud Spanner database operations using Python Client Libraries
Cloud Spanner Introduction demo with examples
zhlédnutí 1,2KPřed rokem
Cloud Spanner Introduction demo with examples
Automate Dataproc workloads using Cloud Composer
zhlédnutí 3,6KPřed rokem
Automate Dataproc workloads using Cloud Composer
Composer - Airflow https operators, Extract data from web API using https operators with examples
zhlédnutí 3,8KPřed rokem
Composer - Airflow https operators, Extract data from web API using https operators with examples
Thanks for your support !!! and More to come
zhlédnutí 91Před rokem
Thanks for your support !!! and More to come
Schedule secured data load from Google sheets to Big query|Cloud Functions,Cloud Scheduler and more
zhlédnutí 2KPřed rokem
Schedule secured data load from Google sheets to Big query|Cloud Functions,Cloud Scheduler and more
Big Query Clustered Tables with Examples
zhlédnutí 4,5KPřed rokem
Big Query Clustered Tables with Examples
Big Query Table Partitions with Examples
zhlédnutí 7KPřed rokem
Big Query Table Partitions with Examples
Google Cloud Storage Object Versioning and Auto Class
zhlédnutí 1,2KPřed rokem
Google Cloud Storage Object Versioning and Auto Class
Schedule Dataproc Workflows using Cloud Scheduler
zhlédnutí 1,8KPřed rokem
Schedule Dataproc Workflows using Cloud Scheduler
Cloud Scheduler Introduction with simple examples
zhlédnutí 4,9KPřed rokem
Cloud Scheduler Introduction with simple examples
Export and Analyze GCP Billing data with BigQuery and Looker Studio
zhlédnutí 3,3KPřed rokem
Export and Analyze GCP Billing data with BigQuery and Looker Studio
Google Cloud Storage (GCS) Object Lifecycle Management with Examples
zhlédnutí 4,2KPřed rokem
Google Cloud Storage (GCS) Object Lifecycle Management with Examples
Dataproc Workflow Templates | Execute Jobs DAG on Managed Dataproc Cluster
zhlédnutí 3,2KPřed rokem
Dataproc Workflow Templates | Execute Jobs DAG on Managed Dataproc Cluster
Big Query Routines , Stored Procedures and Anonymous Blocks for repeated tasks
zhlédnutí 4,2KPřed rokem
Big Query Routines , Stored Procedures and Anonymous Blocks for repeated tasks
Big Query Federated queries | Connect and execute Cloud SQL queries from Big Query
zhlédnutí 1,9KPřed rokem
Big Query Federated queries | Connect and execute Cloud SQL queries from Big Query
Cloud SQL Python connector | Access Cloud SQL Instance Programatically
zhlédnutí 3,6KPřed rokem
Cloud SQL Python connector | Access Cloud SQL Instance Programatically
Welcome back please upload more videos on GCP services and GCP certification related
Your videos are great, kindly maintain frequent videos
I would appreciate your response in advance. How can we identify the path of a specific resource? For example, in the video, you specified the path as: projects/{project_name}/datasets/.... Do we need to grant additional privileges to our default service account for executing the Eventarc trigger?
This is very clear, thank you for this excellent overview.
Is it expensive to use Cloud Composer?
how to contact you
Thank you so much. It helped me a lot to understand difference between native and external table clearly. Thank you so much
Hi can you share sample xl data file this video
excellent Anjan, will you provide online training.
good
Great explanation. I have 1 doubt that which flow is good Cloud scheduler scheduling dataproc workflows Or Scheduling dataproc jobs using Cloud composer? Can you pls help with use case sir
@@ankitsoin5185 2nd option is better in many ways
Hi Anjan, I added test Cvs file in gcs bucket manually but I am unable to open in cloud storage default. Could you please explain why?
@@harinireddy2875 are you not able to open the content of the CSV file ? Did you check the ACL of that bucket?
Yes, access is present
@@anjangcpdataengineering5209 when I am trying add the file which is separated with delimiter , it’s throwing an saying body_21 can not be null and with similar errors. I want add the files how is it and do transformations later. (I am not able on header toggle also .. if I do throwing an error with some error connection.
Thank you for this awesome video, it explained the flow very well. Could you please also upload a similar video where we can loan on premise data in bigquery using composer. If the video is already there, please put the link in the comment.
How to schedule these sir
Thanks for this video. Keep making more such videos
what is this event and where it is coming from?
Hope your health is good, great dedication and commitment Anjan
great presentation, informative session tq Anjan
Thank you! This explanation help me a lot!!
Nice 👍
informative session, excellent presentation
How to do this by terraform please
Excellent usecase. Very helpful
Really good! Thank you so much
its great and systematic presentation of content, highly appreciate your efforts, god bless you Anjan.
Thank you
Nice 👍
Super useful. Thank you for the post~
Is there any way I can find path for csv file (greenhouse_dtls) file to download?
Thank you so much for your video. It has helped me a lot.
Your videos are very good and deserves good reach. Thanks for sharing all these videos.
nicee
I was assigned to GCP project from JAVA development and was told to complete the course and do the certification. Your videos are helping me a lot. Also if you are active now, please make latest tools videos also and also IAM related videos etc. It will help a lot of IT guys like me. Thank you for your videos.
bash_task = BashOperator( task_id='bash_task', bash_command='date:echo 'Hey I am BashOperator' ', ) In above code, I thing dag=dag line is missing Please suggest on this
Thank you
hola, como podria subri un archivo desde local a Cloud storage usando cloud function?, Gracias
I really enjoyed your first video in this series but please, try to keep the intro and the rest of the video the same volume. I had the volume turned up to max since the voice was soo low in the previous video, then i came to the next part and the introduction is EXTREMELY loud compared to the volume in last video. It was a very painful experience for my ears.
I need to connect to a third party API i want to do it via cloud composer in gcp basically airflow is it possible to do it and move the data to gcs how can i do ot
Awesome
Nice and excellent content, and also very well explained the concepts with screen navigation steps. Are you giving any in person or online trainings? Can I get your email id?
Yeah the voice is so bad , couldn't hear most of the words properly
How to do the a ove for 100 Gbs of data that is going to come in the buckets
Very useful, thanks
Sir how to reach out to you?
while creating the DAG i am getting error as SQLite cannot found , it is asking to change the configuration can you please suggest me to what could be done and suggest me the video which i can work on dependencies and fixing the error t
Is it possible to create Partition on already existed table in BigQuery?
Hi @Anjan, I am unable to create a GCP Composer environment. After around 45 minutes it shows the error "Some of the GKE pods failed to become healthy. Please check the GKE logs for details, and retry the operation." I have configured the setting and given the necessary access. I am using composer-2.7.0-airflow-2.7.3. But it failed to create the environment. Can you please help with this? Thanks.
Did you find any solution? I am facing the same problem.
Are you using a trial account? The composer would need more resources that the trial account does not provide
Thank you for providing detailed video. Can you please able provide the dataset for testing ?
Excellent Content and brief explanation
first thank you for this amazing coursers , seconde can you pleas sheare with us the presentation of all this playliste