Twitter Data Pipeline using Airflow for Beginners | Data Engineering Project
Vložit
- čas přidán 29. 05. 2024
- Join Course Python for Data Engineering - bit.ly/4140KI9
This is End-To-End Data Engineering Project using Airflow and Python. In this project, we will extract data using Twitter API, use python to transform data, deploy the code on Airflow/EC2 and save the final result on Amazon S3
Code and commands used in the project - github.com/darshilparmar/twit...
Other resources:
Twitter API - / twitter-api
Python (Installation & Basics) - • Python Tutorial - Pyth...
Setup AWS Account - • AWS Account Setup - St...
Airflow Doc - airflow.apache.org/docs/
Join Data With Darshil Discord Server: / discord
Timestamps
0:00 Project introduction
0:56 Section 1 (Prerequisite)
2:21 Section 2 (Understand basic concept)
7:55 Section 3 (Project execution)
👦🏻 My Linkedin - / darshil-parmar
📷 Instagram - / darshilparmarr
🎯Twitter - / parmardarshil07
🌟 Please leave a LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
3 Books You Should Read
📈Principles: Life and Work: amzn.to/3HQJDyP
👀Deep Work: amzn.to/3IParkk
💼Rework: amzn.to/3HW981O
Tech I use every day
💻MacBook Pro M1: amzn.to/3CiFVwC
📺LG 22 Inch Monitor: amzn.to/3zk0Dts
🎥Sony ZV1: amzn.to/3hRpSMJ
🎙Maono AU-A04: amzn.to/3Bnu53n
⽴Tripod Stand: amzn.to/3tA7hu7
🔅Osaka Ring Light and Stand: amzn.to/3MtLAEG
🎧Sony WH-1000XM4 Headphone: amzn.to/3sM4sXS
🖱Zebronics Zeb-War Keyboard and Mouse: amzn.to/3zeF1yq
💺CELLBELL C104 Office Chair: amzn.to/3IRpiL2
👉Data Engineering Complete Roadmap: • Data Engineer Complete...
👉Data Engineering Project Series: • Data Engineering Proje...
👉Become Full-Time Freelancer: • Best Freelancer Series...
👉Data With Darshil Podcast: • Podcast Series - Data ...
✨ Tags ✨
data engineering project, complete data engineering project for beginners, learn data engineering, data engineering roadmap
✨ Hashtags ✨
#airflow #dataengineering #project
EDIT:
Elon Musk ruined the API part, you can use some other APIs
Here's one of the subscriber built the same project using other APIs - chenmeiqiao.notion.site/How-I-Reach-Out-to-CZcamsrs-I-Like-As-A-Data-Engineer-c37bdddefde54c3789229ffa5a789432
or you can use a static dataset from Kaggle and then use Airflow to process it
www.kaggle.com/datasets/mmmarchetti/tweets-dataset
FAQ:
1. Twitter removed free access: It still has free access but with limits of requests you can make - developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api
2. You need to request for V2 access: You will get an error but just by reading the doc you should be able to access Elevated Access
Worked very hard for this project🤞
Don’t forget to hit the like button and if you want to support my work you can join channel membership, it's only 59rs per month so that I can keep these content for free
already did !!! appriciate your efforts brother !!
Hey! Are you using Elevated access level because Essential doesn't work with the code you are using
Amazing Darshil!! I was able to replicate the same in GCP
@@nishantagarwal8016 Awesome!
Create linkedin post and tag me
@@Soulfulreader786 There is a free version also
this is what is called content and actual do something for community ..rather than just doing faltu things...thank u so much Darshil ❤️
Thank you Sumanta
_"faltu things"_ 😅
Faulty as taking other people interviews 😂😂
@@DarshilParmarredo the project with Twitter api!!!!!! now!!!!!!
My sincere thanks to your efforts for contributing these kind of real time scenario based use case examples. 🙏
one of the best tutorials I've ever seen on CZcams, a real-world example that was really interesting
In data engineering community you are the GEM. Thanks from the bottom of my heart Darshil. Keep growing keep inspiring.
This was a very helpful video for me. I have spent the last year learning web development, and I am expanding my skills to include ETL. This video gave me a great overview of the process.
Mr. Darshil you have saved a lot of time for us by this video. This video has more content than others talkative videos of other youtubers. Thanku so much
I came here seeing your LinkedIn post with comments,Being a fresher I was struggling for a project. This is Absolutely what I was looking for. Just Awsome. Thank you so much for your time and efforts.
Hi Darshil, Thank you for this .It was a great learning experience and it was fun too ! 😛.I am eagerly waiting for more such videos on airflow
You always continue to be great asset in data engineering content, thanks again!
Thanks for creating this project, with the help of this i schedule my data extraction task ,currently i am using solcast API to fetch weather data .
Hey Darshil, thank you so much for this. Of course I got errors but as you said in the beginning, those are part of the journey. Definitely going to follow other tutorials from you as well regarding Airflow and AWS!
Love the way you have explained things. Loads of success to you😇
Great! Got broader understanding of api, airflow and aws. Thank you!
As usual very nice content and appreciate your effort. If you could make a video with OOPs approach and include some test cases that will be really great.
This video is worth more than paid content on many websites.Thanks for such content.
This is really Amazing Darshil. I would also like to see Architecture level videos and how all tools all integrated into the cluster.
Really Appreciate your efforts to the Data Community you are doing a great job. Thank you ❤️
You are welcome
this is awsome !
can't wait for the next data engineering projects, darshil🔥
greetings from indonesia
Thank you for your time and effort to make these videos. Really helpful.
I really appreciate the effort and simplicity..! I was looking for actual content like this ... Thank you Darshil 😇
You are welcome
Amazing Video Darshil, i really appreciate your video, you are a great professor!! Thank you very much keep doing this great content!!
This video is very helpful. Keep up the good work Darshil.
Please keep on bringing content like this it will be really helpful for freshers who want to understand, gain knowledge and shape their career in Data engineering🙏.
I will try my best
Now with the arrival of ChatGPT, even though one can generate CODE without analyzing all these, it is always better to understand the logic, nuance and intricacies of CODING. This vid helps a lot in deconstructing this step-by-step.👌
Please take out time and create deeper conceptual projects like this. If possible please provide real time scenario based projects. We will be really interested to get further depth. This one is awesome!!
Super practical, valuable tutorial. Thank you for sharing, sir.
Awsome brother...Loved the way to teach...Hoping for detailed projects.....thanks a ton brother.....
Really like your approach towards Learning Data Engineering concepts by doing project stuff. A good short smart project to learn basic working of a Data-pipeline using Python, AWS services and Beginner usage of Airflow. Thanks Darshil :)
I have been following u for a long time now I don't even look at others videos......u r just great ....keep up the great work
Thank you, I will
Very good Darshil, keep the good work, cheers from Brasil.
Really love the content darshil.
Very precise explanation.
Thanks sir, I learn a lot following this project guide. I understand more about how to use tweepy, AWS services and airflow by following this tutorial. I hope you always have a nice day.
You are welcome, check out more projects available on channel
Thanku darshil for making this project, please make this type of project on regular basis so that it will help us to enhance our skill
Thank you Aarav, I’ll try my best
Hey thanks man I've been eagerly waiting for this. Would now explore and learn!
Thank you Hritika for always supporting
Darshil, amazing! I do not have words to say thank yoU!
Amazing...got to know something intresting. Thanks for the detail explanation.
Good job mate. I thoroughly enjoy the video. Thank you.
Eagerly waiting for this project. Thank you darshil for such amazing projects...
You are welcome
Excellent job Darshil.
Wowww., Wonderful explanation.., never before ever after...
True master for Big Data ,🙂Darshil
Thank you so much 😀
THANK YOU SO MUCH FOR SUCHA QUALITY CONTENT ......GOD BLESS YOU
Keep up the good work! Your project is the best. Greetings from Chile 😁
This is so impessive. Keep up your good work. I wish for a true success in your life
This is what exactly whay I'm looking for! Thanks brother.
Best content ever..👍
So nice of you..
Really want more such Projects..
I will try my best
Thanks for teaching basics of air flow and Dags..
precise and clear content. Amazing
i m elder than u.. but u r an inspiration bro... i m new to data engineering..
Darshillllllll this is Gold. I literally love you rn 😭
Great and useful video. Thanks Darshil 🙏
Awesome demonstration, truly appreciated l, waiting to see lot more soon
I find it worthy to join a youtube channel membership for the first time. I hope with smart and harwork from myself and your help, I am able to land a Data Engineering job soon.
Bro I am unable to get access to Twitter API v2. Could you please help me out with that part?
Your video was really helpful. Thank you very much
Perfect Explanation .. keep it up .. thank you 🙌
Great darshil this is really helpful video
Thank you.. keep uploading content like this please
Thank you Tejas for always supporting
Great Video Darshil. Keep up the Good Work :)
Thank you👍 for the awesome content We want more data engineering real world end to end project with real time streaming to deal with real world problem and situation.
I will try my best
Great videos, Darshil! Also a side note, I often watch CZcams videos at 1.5. Yours feel faster even at a normal pace. :)
My natural talking pace is faster, many people complained about it but I can’t help it
great content
Thanks Darshil bhai, very cool
thanks for sharing info - A step by step guide
Thank you so much! I encountered some issues along the way, but was able to resolve them!
Bro I am unable to get access to Twitter API v2. Could you please help me out with that part?
@@rohitpandey9920 Yeah, just delete the default and create a new one. Worked after
clear and concise content!! Great!
I watch u everyday ✨ and working on my skills to became a Data engineer 🙂
Real content is finally here ❤ Loved the tutorial.
Bro I am unable to get access to Twitter API v2. Could you please help me out with that part?
Very informative and practical video. Thank you for making a video 😊. Waiting for advanced level video on airflow.
were you able to extract data in csv??
Thank you for the video. It was helpful.
great work you helping lot of people.
Thank you for another great tutorial!!
Awesome sir...... That's what I actually want...
enjoyed the process! took me more than 12 hours to do this lol with all the trouble shooting and error fixing!
can u suggest me how error on " from airflow import dag" error was solved on vscode ??????
Appreciate the content. Cleared the basics by watching it.
What about execution?
@@DarshilParmar Not working
Those who are stuck in the api part can look for some other apis and perform ETL on it too and make your own project.
Darshil helped us to understand the concepts therefore its not mandatory to use only Twitter api.
Thanks a lot Darshil :)
What other APIs are a good place to start other than the Twitter/X API used in the video?
@@snehalsylasmalladi9320 you can search for rapid api or search for open source apis
Very clean and understandable 🎉
Great tutorial, appreciate it, keep up the good work brother. :)
Great explanation. Thanks, Darshil! 😊
Glad you liked it!
Awesome! Thanks for sharing it!
Awesome Work !!!
Thanks and keep doing more... Please also make python part 2 ..
Great explanation! Will try to replicate in GCP
Please do a complete project based on streaming data and transform the data using spark and orchestrate it using airflow and then store it cloud or any database.
Great work!
Thank you soo much darshil sir❤
Wish i found you earlier. i am learning a lot from you 🙂, recommended your channel to all my colleagues that are in data field
Awesome, thank you!
Very crystal clear 🔮 explanation 🎉
Bro I am unable to get access to Twitter API v2. Could you please help me out with that part?
Please keep making more videos like this!
I will try my best
this is perfect, thank you Darshil
Thank you Karan
thanks, very helpful video
Darshil is just amazing 😊 . Good content!
Ah Whatever is happening to the twitter API doesn’t really matter 😂
Hi darshil you are doing a great job,have to request you to create some projects around Databricks and synapse in Azure. We all are waiting with desperation. Likewise AWS,Azure is also hitting popularity and companies ask more for Azure Data Engineers. I know you are very busy creating other playlists. But it's a humble request to create only 2-3 Data pipeline projects in Azure. Those who wants #Azure#Data Engineering. Please hit like so it came at top and Darshil got to know.
Super excited for that series🌟⭐
Yes I see azure in demand.
@@prabhatgupta6415 Yes prabhat in early level of carrer from 1-5 Yrs of experience, Azure is famous.
Mentor of many data engineer ❤
cool man, just what I needed
excellent, thanks for the content
great video. Thank you
Thank you! thank you! thank you!
Very informative👍
amazing as usual
Thanks for the video!
Thank you for everything it helps me so much
Thank you for support Karthik
good video, very detailed