What Is Docker - Docker Intro And Tutorial On Setting Up Airflow | High Paying Data Engineer Skills

Do data engineers need to work with Docker?
It's one of those tools that comes up every now and then.
Both in terms of data engineers wondering if they should learn it also in terms of me needing to use it.
Perhaps its to set-up Airflow via Docker-Compose.
Or just to spin up a basic website.
Docker is, at the very least, good to understand.
Here is the airflow docker set-up
github.com/bAcheron/basic-air...
0:00 - Intro
0:30 - Should Data Engineers Learn Docker
9:46 - Basic Docker Tutorial
14:34 - How To Set-up Airflow With Docker Compose
If you enjoyed this video, check out some of my other top videos.
Top Courses To Become A Data Engineer In 2022
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/​​
Or check out my blog
www.theseattledataguy.com/
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy.substack.com/s...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

[@RahulDhanawade-im7su]

yes, Tech with nana explained docker best way possible. i came here to understand how as a data engineer I can use docker. Thanks again!

[@abel_analytics]

This is really a great tutorial.
Not just jumping into setting up airflow using "docker pull apache/airflow" but building the image from scratch.
I'm also trying to replicate the image but hitting a wall.
Do you mind sharing the "dockerairflow" folder so I can use it as a reference?
The timeline. is 19:06
Thank you

[@maxedouthc]

Love docker / helm / k8 but have not gotten into it with a data pipeline. Would be cool to expand on this video and see high level of docker/k8/py or c# / azure / snowflake

[@ljloaizap]

Hey Ben, thanks for this video, it's been helpful!
I was trying to run exactly same code but ran into an issue with "airflow-webserver / command". I guess it's because I can't see complete line. Would you mind sharing with us the code? Appreciate it! :)

[@emilmoller9644]

What are your thoughts on the data engineer career path on codecademy? How does it compare to other courses?

[@eunheechoi3745]

Do you know how to switch http to https for the web-server? I’ve already configured docker compose with SSL configuration like environment and volume.
Can you please make a video

[@ZoioGame]

Thank you for the quick tutorial. I am a junior data analyst trying to manage better my data pipelines.
Do you think the way you did the configuration there has some security issue? Because I'm going to replicate part of these video at my work and I am always afraid to mess up

[@SeattleDataGuy]

I'd recommend using a managed service like MWAA or Cloud Composer if you're planning to manage this for a company. I am unsure about the scale of your project but Airflow in the wild takes a lot of effort to manage.
If you get to a certain scale, it's just too much for one person to handle and it should be managed by the DevOPs or TechOps team.
Here are a few articles that discuss the challenges and how-tos of managing airflow at scale
shopify.engineering/lessons-learned-apache-airflow-scale
tech.scribd.com/blog/2020/breaking-up-the-dag-repo.html