How to Run a Python Docker Image on AWS Lambda
Vložit
- čas přidán 25. 07. 2024
- Learn how to run a Python Docker image on AWS Lambda using Lambda container images.
Using Docker containers for your Python app on AWS Lambda means you can create a special environment for your app with its own tools and settings, making sure it works everywhere, and also making it easy to include all the things (libraries and packages)your app needs to run correctly.
🛠️ Project Code: github.com/pixegami/aws-lambd...
🔗 Links
Lambda Base Images: docs.aws.amazon.com/lambda/la...
AWS CLI: aws.amazon.com/cli/
AWS CDK: docs.aws.amazon.com/cdk/v2/gu...
Docker: www.docker.com/
📚 Chapters
00:00 Why run Docker on AWS Lambda?
01:04 Create CDK Project
01:45 Create a Python handler app
04:07 Create Dockerfile for Lambda
06:23 Test the Docker image locally
07:11 Create AWS Docker Lambda with CDK
09:44 Deploying to AWS
11:59 Wrapping up
#pixegami #aws
You are brilliant!! Following your steps, I deployed several Python apps to AWS Lambda. Your instructions were clear, code was perfect (worked fine without any changes) and explanations were precise. Very rare in free CZcams tutorials. Thank you.
That's awesome feedback. Thanks for sharing that! Glad it was useful for you.
Dude, I cannot tell you how AMAZING you are and how AMAZING this video is. I could literally hug you through the screen. Thank you so much for your video man. It was a PERFECT explanation.
Thank you! I'm glad it was helpful.
Thank you for sharing this! It saves me a lot of time trying to create a bunch of layers to execute some heavy deps functions on AWS Lambda 🤯🙏
You're welcome! I'm glad to hear it helped :)
How do you even know what we want? Short simple to the point explanation. Thank you so much. Please keep creating such amazing videos!
Thank you, I appreciate your comment, and I'm glad you enjoyed it!
This is was so helpful! Thank you so much!
Fantastic video. It covered so much (CDK, Docker, Lambda) in just 13 minutes. And it actually all works really easily. My first interaction with lambda was a few years ago and decided I would never, ever touch it. But after this video, with CDK and the other progress since then, I am reconsidering moving my workflows over to lambda instead of FastAPI. Pretty much all because of watching this video.
Thank you! I really appreciate this kind of feedback, and honestly it's hearing things like this that motivate me to make this content.
I hope that project all goes well for you!
Perfect! This video is brilliant, If anyone need to Run a Python Docker Image on AWS Lambda. Check this, You will definitely able to solve your issue
Glad it was useful!
I'm very thankful for this video. I really struggle with GUI tools like websites. This command line interface solution really helped me to ship and urgent work I was struggling to configure on aws web interface. Thank you for your work.
Glad it helped!
omg exactly what i needed thanks so much!!
You're welcome!
Great video! Thanks :)
New skill added to my resume, thank you so much :D
Congrats! Well done :)
This is perfect! Great work, thanks!!!
Glad it was useful, thank you!
Thank you for putting so much effort into this video, the quality is just magnificent. Your channel is super underrated, I hope you get the recognition that you deserve soon.
Thank you! Really glad to hear you enjoyed this :)
wow! Excelent video. Thanks so much!
You're welcome!
You've earned a sub my man.
I absolutely needed this bro. Thank you so much. +1 follow
Thank you! Glad it helped :)
Thank you, gained a sub.
Thank you :)
Great. thank you
No worries!
Thanks a lot ! Nice tutorial.
Do you plan to make a video tutorial for a larger project, such as building a full-stack website or something similar ?
Just use docker-compose for this.
Thanks! I do plan some in the coming months, but right now I'm just trying to build a base of smaller videos (like this one) to cover all the basic tools first :)
@@pixegami bump on @davidcasanova2951 's request. Fantastic video
Great video! When creating the CDK project why do you choose typescript as your language and not python? When I choose python as the language the lib folder is created in the venv folder.
I tried Python before and I'm not sure I like the experience as much as Typescript. CDK is natively developed in Typescript, so I feel it makes the quality of code hinting better, error messages easier to read (without all the JSII stuff they used to translate it to Python) and most importantly, it has the largest example base online and in tools like GPT.
hello thank you for your video , i have a small question , in my lamda function i have some api keys ,how to do the .env config in aws with docker
THANKS
You're welcome!
What if we want to use env variables? Where would we place them and how can we retrieve them in the py file?
Great explanation. What are ways you can pass parameters to the container?
Thank you! Well, it depends on what kind of parameters you'd like to pass. Environment variables are probably an obvious and easy way to get started. You could also potentially modify/override the "command" or the "entry point" it is executed with.
Thanks. It could be more helpful if you walk through the permission setting in IAM, as I end up spending an hour configuring that during bootstrapping as a newbie.
Ah I see, thanks for sharing that. IAM is one of those things that I often forget can sometimes be very challenging unless you've used it before. Noted for next time!
fyi for me these issues popped up around the 943 mark (Deploy to AWS). Assuming you are running as a managed user you'll need to add the appropriate policies to your IAM group as 'Permission denied' starts popping up each time you run cdk deploy/bootstrap. I had to add stuff like AmazonS3FullAccess and AmazonSSMFullAccess
thanks for tutorial. What is your terminal that suggest commands?
I use fish terminal. See this video for my Mac terminal setup: czcams.com/video/ugwmH_xzkCA/video.html
king
Would this setup also work to deploy a streamlit app? Or is it only for request-response type apps?
Ah, I don't think it works well for Streamlit. Streamlit relies on a session and an active connection with the user, but Lambda doesn't really do that.
But stay tuned, I'm also working on a tutorial for deploying Streamlit with Docker to AWS as well.
what is the terminal autocomplete feature you are using it's nice how it autocompletes for you
Looks like zsh-autosuggestions
Yup! Or you can use `fish` terminal and get it out of the box. I have a video on the terminal setup here: czcams.com/video/ugwmH_xzkCA/video.html
Is there anything extra I have to do to get environment variables working with this setup?
You should be able to apply environment variables via the CDK construct for Lambda.
See the `environment` parameter: docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda.Function.html
Hii can you make a new video for Docker Python API deployment?
Error saying invalid entry point i am using arm64?? Im using m2 any fixes?
im trying to create a docker image for a python lambda from an amplify app, how would this work for that setup?
I haven't used Amplify, but from what I understand it creates and manages the Lambda runtime for you. I'm not sure if you can extend it to support images. Let me know if you find out otherwise.
That's why for more fine-grained control Iike to use AWS CDK for the infrastructure.
Does the cost of lambdas change with the use of ECR? Or is there any additional cost generated by using ECR?
ECR cost would be billed separately to Lambda. I think ECR also has a free tier? aws.amazon.com/ecr/pricing/
Fantastic video! However, how do we handle logic that takes more than 3 seconds to execute, considering Discord messages have a 3-second limit?
I don't quite get the part on LAMBA_TASK_ROOT? is it an environment variable that u need to set up when u first create the lambda function? the video doesn't seem to cover it..
Is it possible to deploy docker image using terraform? If so, can you do a video on this please?
I'm gonna guess it's possible, but I haven't used Terraform before. I'll add it to my list to check out!
I follow your steps and faild to build on the RUN pip install ... I commented it out and can build the image. What it take to get pip command to work? I use Windows 10 and curl command (in PowerShell) does not work with the URL. What's the ../2015-03-03/... for?
Hmm, what error do you see when you try to run pip install?
@@pixegami I built my image, it runs with no errors but when I do Curl nothing happens. so, as saml asked, what ../2015/... stands for? or where it comes from?
@@mohamedabdo-xg6tw , I did have same questions, here the GPT help me
The 2015-03-31 represents the version of the AWS Lambda API. When you send an HTTP POST request to this endpoint, it triggers the Lambda function handler specified in your Docker container.
Hi, this setup spike my billing very high,
The setup was to build lambda function to read the latest file from the s3 dir and make transformation then finally to s3 target dir,
So this all setup with the python script has to run once the s3 notification to lambda function that an file just came to s3.
But it went into a loop and made the s3 and lambda billing spike
Let me knew what is the issue in my setup that i didn't noticed at first while running this python script in lambda
Uh oh. It sounds like you have a recursive Lambda function. I guess after you update the S3 file, it probably triggers another event (which triggers the function) in a loop.
Normally, if you want to trigger Lambda from an S3 event, I wouldn't recommend to update/write back into the same S3 - maybe just use a different bucket for the output.
For the billing spike, you can create a ticket to AWS support to explain the mistake, and see if they'll be able to give you a refund on the cost (it usually works if it's the first time you are making an honest mistake).
I came here after trying to install my libraries to my python lambda function and it give up so I am trying to upload it as a docker image
Yup, uploading a library with Python *without* Docker is surprisingly difficult, especially if you want to use libraries that have an OS-dependent library under the hood (which includes AI/ML libraries, numpy, and more). That's why the Docker way is really good to know :)
Amazing. I'm facing an issue in Dockerfile in "RUN pip install -r requirements.txt". when I run cdk deploy, I get Failed to build asset error. In local, it is running correctly. Any idea how to resolve this?
meanwhile I commented 'RUN pip install -r requirements.txt' and returned just dummy message without numpy. This time Cdk deploy worked and the url was giving me the json response
Hmm, I think we'll need to see what the actual error is - the build could fail for a variety of different reasons.
does not work for me, my curl command gets stuck without any response
Have you run the container successfully? Do you see any output from the container terminal?
Trying your tutorial on windows, prompted the error: 'curl: (52) Empty reply from server'. Any help will be apreciated!
Is this for when you are testing it locally? What about the Lambda docker output itself? In the terminal where you are running the image locally, do you see any messages?
Solution : on M chip Mac : change credstore from desktop to osxkeychain , use sudo nano ~/.docker/config.json
Thanks for sharing but I’m getting the error
ERROR: failed to solve: public.ecr.aws/lambda/python:3.11: error getting credentials - err: exit status 1, out: ``.
Can you please advise a solution ?
Hmm, sounds like a permissions issue. I recommend checking how you are setting up your AWS credentials for this. AWS goes through a series of steps to try and determine your credentials: docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html#credentialProviderChain
I wish real world senerios are this simple lol, I am currently using node and canvas and that's needed a lot of Os level dependies like gcc-c++ cairo-devel etc, how to cache them because everytime I make changes I need to build docker image again and that is taking too much time
Well, tutorials and examples are always going to be a simpler cleaner version of real world problems which does have its own challenges :)
But your problem does have a solution. Docker builds images in layers, so anything you want to update/change, just put them in a later layer (e.g. lower in the Dockerfile) so it doesn't have to rebuild everything up until that point.
That way it'll just re-build your changes into the image without having to go through the dependencies installation again.
If you want to take it a step further, check out using GitHub actions to automate your workflow and deployment (it also has caching): czcams.com/video/9uMcN66mfwE/video.html
Nice video. Thanks @pixegami
Thank you!
07:02 it's interesting that I've done the same with Postman and it didn't work. CURL works, though
Hmm, I wonder why. Does it throw an error?
On the pre-requisites. How to install cdk on aws 2023 linux image.
sudo yum install nodejs npm
sudo npm install -g aws-cdk
sudo npm install -g npm@10.2.5
sudo npm install -g aws-cdk
Dumb question, what terminal are you using?
On the pre-requisites. How to instal Docker on aws 2023 linux image.
sudo yum update -y
sudo yum install -y docker
sudo systemctl start docker
sudo systemctl enable docker
sudo groupadd docker
sudo usermod -aG docker $USER
docker version
Thanks for sharing this!
After the above you may also have to restart your machine before commands such as docker ps or similar can succeed in connecting to the daemon.
Great video, thank you! Wondering how this would work if using poetry instead of requirements.txt? I've seen a lot of different answers on SO and yotube.
It's not python it's typescript
The Docker image is a Python app. If you mean the CDK - yes that's Typescript. You can use Python for it too, but I generally go with Typescript since it's the 'native' language for CDK.