Amazon Books Data Pipeline: End to End Data Engineering Project using Airflow

Hey Data Engineering Enthusiasts!!
In this video we will be building an ETL data pipeline using Apache Airflow. This pipeline extracts data engineering books from amazon, and stores it in Postgres Database. The pipeline runs on a schedule and pulls data from the website.
This video will help you build a basic data pipeline and also get a repository of data engineering books, all at the same time.
Github link: github.com/sunjana2199/amazon...
Timestamps:
00:00 - Intro
01:00 - Pipeline Design
03:16 - Install Airflow
04:49 - Install PGAdmin
05:44 - Create Books db
06:45 - Create Postgres connection from Airflow
07:27 - Build DAG
09:32 - Define functions
10:43 - Add Tasks
11:26 - Dependencies
11:48 - Manually Trigger DAG
12:14 - Query data on PGAdmin
12:42 - Conclusion
Links:
Airflow Documentation
- airflow.apache.org/docs/apach...
Code for PG Admin:
"""
postgres:
ports:
- "5432:5432"
pgadmin:
container_name: pgadmin4_container2
image: dpage/pgadmin4
restart: always
environment:
PGADMIN_DEFAULT_EMAIL: [e-mail]
PGADMIN_DEFAULT_PASSWORD: root
ports:
- "5050:80"
pgadmin:
container_name: pgadmin4_container2
image: dpage/pgadmin4
restart: always
environment:
PGADMIN_DEFAULT_EMAIL: [e-mail]
PGADMIN_DEFAULT_PASSWORD: root
ports:
- "5050:80"
"""
Hope you enjoy this video :)
Let me know in the comments about what you think of this video!!

[@ashishpasupureddi]

I’m impressed by your work!!!

[@GyanSparsh]

Beautiful, thank you for sharing...

[@anishnair45]

Hey Sunjana ! Thanks for sharing this but for someone like me who is a complete beginner, it's hard to follow along. Is there a step by step approach ?

[@vinodsagar2412]

Hello madam,
Thank you for such a good knowledge on above skills for data science projects.
Madam please do one end to end project using AWS cloud.
Thank you

[@ankitsaxena565]

Dear Mam, what are the skills required for the become data engineer

[@Piyush-xc2td]

you are using some kind for filter or LUTs, looks synthetic

[@brienaustinclayton]

Can you make a video of all free resources for Data Engineering ?