How to Create and Use Virtual Environments in Python With Poetry
Vložit
- čas přidán 9. 07. 2024
- It can be tricky when different packages in Python don't play nice. The solution to this problem is to create an isolated, virtual environment. In this video, I’ll show you how to set up a virtual environment with poetry, a package manager for Python that simplifies dependency management and project packaging.
🚀 Next-Level Python Skillshare Class: skl.sh/3ZQkUEN
✍🏻 Take a quiz on this topic: www.learntail.com/quiz/kkmnju
💡 Get my FREE 7-step guide to help you consistently design great software: arjancodes.com/designguide.
💻 ArjanCodes Blog: www.arjancodes.com/blog
🎓 Courses:
The Software Designer Mindset: www.arjancodes.com/mindset
The Software Designer Mindset Team Packages: www.arjancodes.com/sas
The Software Architect Mindset: Pre-register now! www.arjancodes.com/architect
Next Level Python: Become a Python Expert: www.arjancodes.com/next-level...
The 30-Day Design Challenge: www.arjancodes.com/30ddc
🛒 GEAR & RECOMMENDED BOOKS: kit.co/arjancodes.
👍 If you enjoyed this content, give this video a like. If you want to watch more of my upcoming videos, consider subscribing to my channel!
💬 Discord: discord.arjan.codes
🐦Twitter: / arjancodes
🌍LinkedIn: / arjancodes
🕵Facebook: / arjancodes
📱Instagram: / arjancodes
♪ Tiktok: / arjancodes
👀 Code reviewers:
- Yoriz
- Ryan Laursen
- James Dooley
- Dale Hagglund
🎥 Video edited by Mark Bacskai: / bacskaimark
💻 Code example by Henrique Branco: / henriqueajnb
🔖 Chapters:
0:00 Intro
1:24 Code example
2:00 Virtual environments
3:00 Tools
4:15 Why poetry?
5:20 Creating a virtual environment with poetry
8:33 Activating it
8:55 Installing dependencies
9:38 Deactivating it
10:11 Removing it
10:27 Packaging and publishing using poetry
11:58 Things to watch out for
14:18 Outro
#arjancodes #softwaredesign #python
DISCLAIMER - The links in this description might be affiliate links. If you purchase a product or service through one of those links, I may receive a small commission. There is no additional charge to you. Thanks for supporting my channel so I can continue to provide you with free content each week!
💡 Get my FREE 7-step guide to help you consistently design great software: arjancodes.com/designguide.
There are some additional super useful functionally worth mentioning:
1) In the toml file, there are special characters that define/limit how to update dependencies versions (^, ~);
2) the lock file “locks” the exact versions for dependencies and transient ones. This ensures reproducibility and is a game changes for production ready code. it is possible to test updating packages and only committing the new .lock file if everything works.
3) poetry has a cli tool to auto bump the package version (major, minor, patch) which makes it even easier when managing a package
4) for some compatibility cases, it is also possible to export the dependencies into a requirements.txt file (for example, when using serverless framework to deploy an AWS Lambda function using the pip-requirements plugin)
Poetry is a 10/10 project tool IMO. Im glad I was able to convince my team to move over to it.
same here, never look back at setuptools 😀
Unfortunatelty they are introducing the horrible idea of range-pinning / upper-pinning their abstract dependencies.
This will create a giant dependency-hell of false positives.
It’s been a game changer for me.
What was your team using before?
This has to be one of the best and most concise tutorials I've watched lately. I've been using virtualenvs and venvs with pipfiles for years, but this looks like the next logical step. Thanks!
Great to hear!
I don't think it can be underestimated what a "rabbit hole of pain 🙂" this topic can be for the python programmer who is moving beyond the basics (I know from personal experience lol). Especially compared to `cargo` in Rust. Poetry has simplified my Python development, and this is an excellent video in getting started with it. Thanks for all your great videos!
I find that the most critical part of installing packages for some repo or another is getting the python version right. Very often if pip install -r fails, it's because you need an older version of python. So delete your virtual environment and recreate it with an older version. I usually use conda, so it's conda create -n python=3.7 or so.
This is a great video to understand the use of poetry. After this video, I find the official Poetry documentation making a lot of sense. Thank you!!
Finally sounds like theres a decent python project package manager. Can’t wait to use it
Thanks Arjan, was waiting a long time for this video. Finally Poetry gets explained in a good and clear way!
Thank you, glad you liked it!
I've actually been using conda for this, even for non-Python projects, and it worked well for me so far
I'm using pyenv to manage python versions, and then having poetry creating creating a virtual env inside the project. Really glad with this setup.
At work we started using poetry, it helped us a lot to deploy and also manage de dependencies.
So glad this video finally came out!!
Haha, yes it took me a while :)
Thanks, before watching the video I struggled at quite some points. Now it is very clear to me and I see the advantages of using Poetry :)
Thanks for this!! I started playing with Poetry a little while ago and needed this video!
You're so welcome!
This was really well explained.
you're a champ Arjan
Good video! I've been using poetry for some time now, and recently ran into the case where you have a poetry project that uses a python version that needs to be updated, which means you have to manage multiple python version in your system and need to tell poetry to go from one version to another. That would be a really good topic for another video as I see a lot of confusing information online about pyenv and poetry working together
Try out tox to support multiple
Thank for the video Arjan, I think some medieval emperors would be interested in the practical applications of a language like Python, such as its ability to automate tasks and process large amounts of data. They would appreciate its ability to create visualizations and simulations, which could be useful for military planning or other strategic purposes.
Poetry is life! Seriously it's a great tool, at word we use it all the time and for my personal projets I would use it without an hesitation too!
Love the videos & quality work you put into them. Could you at some point do some examples, such as: wave function collapse, etc ...?
Thank you son much exactly when I tart learning poetry😊 excellent tips to delete old env and reinstall them evently to limite size of project
Thanks a lot for this video. Loved it :)
Glad you liked it!
I love poetry + pyenv. It allows me to define the dependencies exactly so that anybody that clones my repo can get the exact experience I get.
How were you able to have a terminal as a tab in this video on vscode?
Another great video!! Just a few questions:
1. Could you share your shell setup? The autocomplete is amazing!!
2. There doesn't seem to have a seamless way to build the dependencies into a docker and make it as lightweight as compared to building the dependencies through requirements.txt.. How would you suggest going around this problem?
You can use poetry export to generate a requirements.txt based on the dependencies in your poetry.lock file
About the autocomplete: it seems Fig, and you can do some magic AI tricks
@@mattiapiccinetti4301 Sadly, macOS only right now :(
That .venv tip can be handy for people using VSCode's dev containers feature. If you need to rebuild the dev container you end up trashing its filesystem usually, along with any Poetry virtualenvs that may have been created in the container filesystem. By using .venv within your repo's source directory (probably want to .gitignore it), you can rebuild the dev container without wiping your venv, saving you from doing poetry install repeatedly.
You mean mounting the .venv directory into the container?
Thanks for the video. If I recall correctly, Poetry also allows to have multiple sets of dependencies, for example a dev set where you put all your tests dependancies (like pytest), and a prod set for the app dependencies.
Yes that's right!
3:04 venv (built-in)
5:13 lock file: yeah conda also uses that. but i dont understand what does that entail practically.
8:20 idky but in default github codespaces (running ubuntu); via python's builtin venv: the .venv dir was actually only 25 MB despite the packages being large 100-200 MB. and also, on rm -r .venv and reinstalling; it didnt download but used from cache.
I thought maybe the packages are at a global location there.
coming today back to this quick rununp. the extension for reStructuredText lextudio.restructuredtext on vscodium mentions in #409 that
_"While many ... , Pipenv shows a good enough architecture and a clean way to integrate. However, venv is widely used and part of Python standard."_
and looking at pipenv's pypi listing, it seems really good: brings _most_ benefits of conda lineage here:
* manageable as a separate app (via pipx)
* lock files
* separate abstract dependency declarations from the last tested combination
* dependency graph
* Automatically install required Pythons, if pyenv or asdf is available.
* Automatically adds/removes packages to a Pipfile when they are installed/uninstalled
i had tried micromamba¹ (from conda lineage) to install c++ based dlib² package (which₂ doesn't distribute binary .whl on pypi). but it₁ was hard to configure properly, and a bit opaque on where are its things going. there's even no official way to uninstall it lol.
benefits of poetry over pipenv seems that it can be used to:
4:53 * specify python version, so no need of pyenv?
5:04 * publish packages, so no need of setuptools? yeah, the build system table here
5:57 * dev dependencies are mentioned separately :)
6:57 * get info about project. it is even shows python implementation used, so, it hints that it's compatible with other implementations 😋?
things to note:
13:21 python packages using c-extensions (numpy, etc). yeah see, the dlib thing - but yep, i was already aware that this would be issue in any project manager which is python's native.
Does anyone know what terminal setting he is using to get that green arrow and short clean path?
hey can you tell how do you use intelli sense like code suggestion in your terminal when you type python commands ?
2 questions:
1. which config/plugin/script are you using to provide those neat autocompletions when typing terminal commands?
2. what is the advantage of poetry over pipenv? they look very similar in functionality (aside from publishing to pypi), and pipenv creates Pipfile, which looks almost identical to pryproject.toml (same with their lockfiles).
2. I have used Pipenv before and switched to Poetry because it's much faster
Poetry locks dependencies faster, and clearly explains when there is a dependency conflict.
what terminal do you use? its really handy to have autocomplete feature in terminal
Just curious. Do poetry and dependabot work well together? I've read conflicting reports. I tried it on a test repo, and it seemed like dependabot recommended radical version changes in both pyproject.toml and poetry.lock, where poetry said "No dependencies to install or update"
@ArjanCodes, is that Python tab a plug-in console for VSCode?
can Poetry config in project setting be a default when setting up future projects?
Another fantastic video! Thank you 🙂
Glad you enjoyed it!
I started with Python a year ago and came across pipenv and use it since then. Is there an option like the --dev in poetry to install packages only in dev environment? For example, I install mypy and pytest only in dev but not when I run it in prod.
Personally, I've not had much luck with poetry. It isn't deterministic. Sometimes it breaks in bad ways, and this has happened enough in a team environment that I've given up on it completely.
I'm looking for something other than poetry. Currently just using virtual environments, but I am waiting for the right solution to come along.
I ran into an issue the moment I installed poetry. I'm pretty new to Python so I don't know what the issue is but no matter if I try in bash, powershell etc all poetry commands are not found. I tried adding it to the $PATH and nothing works. Any help would be appreciated but nothing has worked for me.
I tried it again on another computer (both use the Microsoft Python) and it worked perfectly but just doesn't work on this other laptop.
I moved now from 1.7.1 to August 2023 version 1.6.1 and it worked smoothly
For a quite big project we are going to build about 50-100 packages, some larger or more complicated then others. Would it be a good practice to create a project and a venv. for each package we build? And is there a good way to host them privately, and have a way to use these packages to build our applications
Thank you for making this awesome video.
Glad you enjoyed the video!
How do you deploy an application built with poetry, e.g. what do I have to do if I want to containerise it?
I saw almost 0 real pros to use some wrapper over pip. What poetry is actually do beside just adding reqs to file for you which pip won't be able to do?
why do you prefer to have the environment inside your project?
but if you want to share the same virtualenv across multiple projects, then is it better to keep it at default?
Given that creating a virtualenv inside of each project ends up taking up more disk space, would it be possible to have multiple projects share the same environment? If so, where do you recommend to put the environment in that case?
nice vid, what do you use for the terminal intellisense
If I have sub modules that have their own packages that need to be installed (thing of plugins). How will we manage installation of these with Poetry?
@arjancodes great video as always. I'm curious about how you have your terminal set up to get context aware autocomplete suggestions? E.g. the drop down like prompts that describe the rest of the commands and flags?
It's Fig
yooo what terminal is that that gives you suggestions like that?
Thank you for this great intro! Unrelated: how are you getting that autocomplete tooltip in your vscode terminal ?
how your terminal suggests all arguments? I've already installed zsh and oh-my-zsh, but it does not suggest options.
Thank you for this video. Do you know how to install / compile prophet using any of the virtual env tools? (for python 3.11.2)
I have been using pipenv for a couple years. I really like poetry so maybe it's time to learn a new tool. It's actually not that different than pipenv, but cool. Thanks.
Yeah I'm fond of pipenv too but it's so slow... 😱
Can export a requirements.txt file for backward compatibility/docker containers:
`poetry export -f requirements.txt --output requirements.txt --without-hashes`
what is the benefit of having the virtual environment folder within the project directory?
Very good. Thanks for sharing your knowlage with us.
My pleasure!Thanks for your comment :)
Wow.. Made it look so easy
Thank you very much
Glad the video was helpful!
Great Video, just wanted to know if poetry is capable of managing multiple dependency files like we de have different requirements.txt from dev,stag and prod. If that is supported would love to switch to it.
I am a beginner. how did you arrive in the environment(IDE OR FOLDER) you are using for installing poetry?
Thanks for explaining! But docker looks more preferable to me in such cases. Poetry could be, if it were not as complicated for set up.
How do you manage multiple python interpreters with poetry? how do you install them and make sure, they don"t interact with the system installed interpreter?
should poetry be installed in an venv or globally?
Thanks for introducing me to Poetry :)
You are very welcome! :)
Very cool! it's like Cargo but for Python :)
Thank you for the video. It is better than Poetry's "Introduction" and "Basic Usage" pages. They are confusing.
I'm glad the video was helpful!
I used Poetry for a while but switched to PDM lately. It's pretty much the same experience but follows the Python PEP standards a little better (though has to sacrifice the nice version number operators), and I hear the dependency resolver is faster in many cases though that hasn't been an issue for me.
Hey mate, how's your experience with PDM since then compared to Poetry? I am trying to learn either of the two and saw your comments, hope you could shed some light. I found poetry is more popular (for now) and hence would get more community support. If there is no noticeable difference, I might just pick poetry.
@@alsonyang230 I still use PDM and I wouldn't go back to Poetry. I'm also just today considering converting an older Poetry project into PDM. But I don't have any very strong recommendation either way. And I'd say that if you learn one, you'll get the hang of the other one fairly quickly if necessary.
@@OlliWilkman Thanks mate, appreciate your input
I like pip-compile (from pip-tools) for the simplicity and keeping with the de facto requirements.txt file.
VSCode was never recognizing my poetry env unless i’ve added it manually into settings.
Hello Arjan!
Thanks for your video!
I would like to know what is the difference between "pip install poetry" and installing with curl (from official docs)? See the difference when working with pyenv?
Follow the official documentation for installation. Rationale: it is better isolated from your python installation. There are scenarios where poetry will uninstall itself if it was installed through pip, because it sees poetry installed even though it is not a dependency of your package.
Poetry in general removes packages that are not a dependency of your package. I combine poetry with pipx. With pipx I install dev tools like black, mypy, pylint, pycln, etc. In that way they are out of sight for poetry, but I can still use them.
@@bartman060890 Thanks for such a detailed answer! It makes sense.
What is the advantage of combining pipx and poetry instead of using --group dev?
@@import_this that would work too! I prefer this approach as it extends nicely to all my python work, also if I don't use poetry. But by all means experiment and see what works best for you :)
@@bartman060890 I will :)
Poetry does not copy TCL into the virtual enviroment for use with tkinter. Is there a fix for this on windows
I use Pycharm. A main reason is how it creates a new environment for me for each new project. VS Code needs an easier built in feature like Pycharm.
pycharm was my favorite until vscode's pylance started giving better code hints and i got attached to some vscode extensions. But there are many features I miss... the doc-string rendering on hover was much neater in pycharm.
May I ask whats the type hinting in your command line? :)
perfect thanks
Glad the video was helpful!
Hi great tutorial, although can anyone tell me how is the auto complete working in his shell, it looks quite amazing and useful.
Hello, love your videos!
Maybe there is something I'm missing, but if I'm already working well with conda, is there any reason I should switch over to Poetry?
there is no reason to switch even from pip actually
Hello!
I love your videos!
Only one thing I would like to point out (in case it happens and you see this comment) is to use a bigger screen? or zoom out a bit?, because it looks really crowded now (watching on PC though)
Thank you for your content!!
Thanks for the tips!
nice content
As always, thanks Arjan for your excellent content! I've not tried Poetry yet but I can't yet see a compelling reason to do so vs using conda/mamba (I usually use the latter). It does look pretty easy to use, but not any easier than conda, just different. What am I missing about Poetry? It does have its enthusiasts here in the comments. How is it's spec easier than a conda env spec?
Having .venv in the project folder seems ok, but I assume you gitignore it and recreate it as needed so it doesn't clutter things up?
What I've done using cookiecutter is to setup a project template with simple make machinery that does conda package generation running 'out of the box' so it's easy to deploy the project as a conda package to a local channel (or other). It creates a project conda env with the project with a basic set of packages for dev, incl. sphinx and pytest. It also does sphinx docs and comes with an initial set of strawman doc files. The make also maintains a conda env spec yaml file so I don't bother with requirements.txt and other devs can recreate the env on their platforms when we share the project through a local GitLab (or just a local bare repo). If I need to deploy a wheel or source package the template has all the bits to do that as well (our use cases do not involve deploying to PyPi or other public repos).
A consideration for us is that we use conda to set envs in which analysts do interactive Python work for operations. In future we might need shell scripts that specify which env to use when invoking particular versions of tools. Some analysts don't even use conda directly, it's invoked in setup scripts to set the correct env or their path is just set to the correct, shared env location when they log in. I guess this is more of the data-sciencey use case that conda is intended for!
The most painful thing about Python packaging is the ever evolving 'how to do' Python packaging question. What I have in that cookiecutter is a bit of a frankenstein between the setup.py, setup.cfg, the new kid on the block pyproject.toml, and the conda-recipe meta.yml. (I need to watch your recent vid about the PyPi way...)
how about the PDM
What are the differences between poetry and pipenv? I don't see many based on this video, maybe just publishing packages on pypi.
i wish you had talked more about the lock file - that is what always messes me up with poetry!
Thanks for this video. I just have a small question. What do you think about `pyenv virtualenv` VS `poetry`?
More specifically, here is what I would like and would appreciate your help --
1. I really like `pyenv`. I use it to install different python versions and **also to install virtual environments**. I don't want poetry to replace it.
2. However, I would like to install packages using poetry so that it automatically gets added to `.toml` file generated using `poetry init` command.
However, I faced an issue in the second command. When I did `poetry add numpy`, it says `Failed to unlock the collection!`.
I am completely new to poetry. Could anyone help me out?
Great vedio. how can we debug the poetry app from VSCode ?
Hello @ArjanCodes, great jvideo about poetry! Thanks! I see that you are using a Python's shell??? What kind of magic is that? Is it an extension of VScode? Thanks man! Have a great day!
How did u make the command completion menu in terminal?
it is secret not for you ;)
What are you using on the shell to get those completions? That look really nice
If you go to the poetry page, there's actually an installation option to get Shell completion for whatever shell you use
But what do you think about pdm?
The metaverse joke was fire 😄
I adopted 'hatch' for package builds because it claims to more closely follow modern PEP standards, but after this video, and reading the comments that mention PDM, I am very confused about the best tool to use :S
Arjan is it okay if I link to this video as a reference in a post I am writing about virtual environments (venv, renv, jail-rooted servers, etc.)?
Thank you for the excellent channel!
Sure Dave! You can link it as a reference :)
7:04 You can do it by simply writing... (watching in the notes to find the command) ...
in 2023, use docker/container or still virtualenv?
12:16 You mention if you're using "pipenv" packages won't automatically be added to a requirements.txt file. I'm guessing you meant to say "pip", since with pipenv dependencies are added to the Pipfile and Pipfile.lock.
Can you make a video about Poetry's competitor PDM and compare the two?
Ditto the newest kid on the block, Rye. Same dev as Flask
poetry lock
poetry update
Much simpler than a lot of other managers
Poetry rocks!
A lot of new repos don't have a requirements.txt file. Instead there's a .toml file. So we need to use poetry instead.
Thanks for the video! For me poetry was a love at first sight ❤ and I use it in every personal project ever since (and often at my work too 😉) A question: what autocompletion for poetry commands do you use here?
It has native completion support built in. Run “poetry help completions” and follow the steps.
Im having issues now , it says its not compatible with my python version: s
poetry + poethepoem = easy python development ❤