Installing Python packages in 2019: pyenv and pipenvGioele Barabucci,
The way Python packages are installed and managed used to be quite convoluted with multiple conflicting alternatives coming and going every few months.
As of 2019, thanks to the efforts of many groups including the Python Packaging Authority (PyPa), the situation is now much better and two clear winners are emerging: pyenv and pipenv. Similar names, completely different tasks.
<ruby-appreciation-moment>Both projects are heavily influenced by
their Ruby counterparts:
The objective of this small guide is to describe how pyenv, pipenv and various other tools work together to install and manage Python packages. This guide is aimed at those who would like to have a look behind the scenes to understand how modern Python packaging tools work together.
In your Python program you use foolib, a library that is not installed
Not a big problem, just install it via
pip install foolib, right?
Not so easy. Here is a couple of issue with that.
Do you have enough rights to install that package? Maybe you need to use
sudo pip. This sounds wrong.
Your application needs version 2.0 of foolib. Another Python application in you system needs version 1.8. Oops, API conflict.
Other people that will install your application will need foolib. You need to document it somewhere, for example in
requirements.txt. But using
pip freezewill record the exact version of foolib you happen to use right now. Any version of the 2.x series would be OK, but there is no easy way to tell this to pip.
Actually all this is moot because you need native datatypes and they are available only in Python 3.7. Unfortunately your Linux distro only ships version 3.5.
All these problems could be solved in many different ways.
As of 2019, the state of the art in installing packages and managing dependencies in Python applications consists in using pyenv and pipenv.
In the following sections we will take a step back and have a look at the tasks carried out by pyenv and pipenv, as well as peeking under the hood to understand their relation to other tools like the good ol' pip and virtualenv.
The pieces of the puzzle
If you just want to install and use Python packages, there are only two tools that you should care about and use directly: pyenv and pipenv. However, to better understand how things work, you should also know a bit about two other tools used internally by pipenv: pip and virtualenv.
Here is a small description of what each of these tools does.
- allows you to install and use specific versions of Python and
its related tools.
In particular, pyenv allows you to install many different versions of Python on the same system and by non-root users. Specific versions can then be chosen at runtime. pyenv plays with environment variables (
$PATH) and symlinks to provide you with a specific version of the
For example, you can run
pyenv shell 3.7.2and, from that point on, whenever you run
python, the executable for version 3.7.2, installed under
$PYENV_ROOT, will be used instead of the whatever other version of Python is installed in your system.
- fetches and installs Python packages.
More precisely, it installs packages into an existing Python installation, for example in
/usr/lib/python3/dist-packages/. If pip is run inside a pyenv environment, it will install the packages into the currently enabled Python environment, somewhere under
- tricks pip into installing packages into arbitrary directories
virtualenv allows you to keep the Python packages of different projects isolated from one another. This is helpful, for example, if different projects require different incompatible versions of the same package.
- is the thing that puts all these tools together.
It allows you to specify in a project-specific
Pipenvfile the direct dependencies of your project as well as the required Python version.
pipenv has two main phases or modes: installation mode and runtime mode.
During the installation phase (when you run
pipenv install), pipenv takes care of:
- finding the best set of dependencies that fulfils the project's requirements in terms of versions and compatibilities.
- using virtualenv to create a project-specific package directory where the dependencies of the project can be installed.
- calling pip to actually install these dependencies.
At runtime (when you run
pipenv run COMMAND), pipenv takes care of:
- using pyenv to create a runtime environment with the specified version of Python.
- integrate the installed dependencies into the current environment.
The big picture
- takes care of the
pythonbinary and all related tools. It stores everything under
- takes care of
- calculating the complete set of dependencies;
- (in installation mode) telling virtualenv and pip where to install the dependencies;
- (in runtime mode) making available an environment with the right version of the Python interpreter (via pyenv) and the right set of packages (via virtualenv).
The only commands you should care about are:
pyenv install VERSIONto install a new python interpreter.
pipenv installto install your project's dependencies.
pipenv shellto enter an environment set up as described in
pipenv run COMMANDto run a single command as if it were inside that enviroment.
An example in practice
Let's see how things work together with a small example.
We are developing a Python application in
We already installed and set up
pipenv (discussed in the
This is the code of our wonderful application:
#!/usr/bin/env python import crayons from pyfiglet import figlet_format print(crayons.blue(figlet_format('Hi there!'), bold=True))
As you can see, we use a couple of non-standard modules.
We declare these dependencies in the
[[source]] url = "https://pypi.python.org/simple" [packages] crayons = "*" pyfiglet = "*" [requires] python_version = "3.7.2"
At this point we run
pipenv install to install all the needed
The output of
pipenv install will show us how the various
pieces of puzzle fit together.
(Note: The output is slightly redacted for the sake of brevity.)
Creating a virtualenv for this project… Pipfile: $HOME/Projects/exapp/Pipfile Using $PYENV_HOME/shims/python (3.7.2) to create virtualenv… Running virtualenv with interpreter $PYENV_HOME/shims/python Using base prefix '$PYENV_HOME/versions/3.7.2' New python executable in $HOME/.virtualenvs/exapp-nXCyFRyd/bin/python Installing setuptools, pip, wheel...done. Virtualenv location: $HOME/.virtualenvs/exapp-nXCyFRyd Pipfile.lock not found, creating… Locking [dev-packages] dependencies… Locking [packages] dependencies… Updated Pipfile.lock (a0242d)! Installing dependencies from Pipfile.lock (a0242d)… Installing 'colorama'▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/3 — 00:00:00 $ ['$HOME/.virtualenvs/exapp-nXCyFRyd/bin/pip', 'install', '--verbose', '--upgrade', '--no-deps', '-r', '"/tmp/pipenv-hhv176qm-requirements/pipenv-9zs0b2h7-requirement.txt"', '-i', 'https://pypi.python.org/simple', '--require-hashes'] Installing 'pyfiglet'▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/3 — 00:00:00 $ ['$HOME/.virtualenvs/exapp-nXCyFRyd/bin/pip', 'install', '--verbose', '--upgrade', '--no-deps', '-r', '"/tmp/pipenv-hhv176qm-requirements/pipenv-zaq1e563-requirement.txt"', '-i', 'https://pypi.python.org/simple', '--require-hashes'] Installing 'crayons'▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/3 — 00:00:00 $ ['$HOME/.virtualenvs/exapp-nXCyFRyd/bin/pip', 'install', '--verbose', '--upgrade', '--no-deps', '-r', '"/tmp/pipenv-hhv176qm-requirements/pipenv-n9xxkvj7-requirement.txt"', '-i', 'https://pypi.python.org/simple', '--require-hashes'] 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 3/3 — 00:00:02 To activate this project's virtualenv, run pipenv shell. Alternatively, run a command inside the virtualenv with pipenv run.
Here is a breakdown of the main steps performed by pipenv:
- pyenv is used to select the version of Python specified in the
- A virtual environment specific to our project
exapp-nXCyFRyis created. (The name is created by hashing the absolute path of the
- The dependency graph is first calculated and then recorded in the
Pipfile.lockis a story for another time).
- pip (the one linked in the virtualenv and provided by pyenv) is used to install the dependencies (including the dependencies of the dependencies), one by one.
At this point the environment where our application could run is ready, but it is not active yet.
We have two options to run our application in the prepared environment: launching a single-shot command or opening an interactive shell.
The most direct way is to run the application as a one-time command.
Pipenv will load the environment for us, run the application and then
exit the environment.
We can do this using
pipenv run ./exapp.py.
$ pipenv run ./exapp.py _ _ _ _ _ _ | | | (_) | |_| |__ ___ _ __ ___| | | |_| | | | __| '_ \ / _ \ '__/ _ \ | | _ | | | |_| | | | __/ | | __/_| |_| |_|_| \__|_| |_|\___|_| \___(_) $
The second way is to open a shell inside the environment from
which we can launch our application or execute any other command.
To do this we use
$ pipenv shell Launching subshell in virtual environment… . $HOME/.virtualenvs/exapp-nXCyFRyd/bin/activate $ ./exapp.py _ _ _ _ _ _ | | | (_) | |_| |__ ___ _ __ ___| | | |_| | | | __| '_ \ / _ \ '__/ _ \ | | _ | | | |_| | | | __/ | | __/_| |_| |_|_| \__|_| |_|\___|_| \___(_) $ python --version # inside the environment Python 3.7.2 $ exit $ python --version # outside, in Ubuntu 16.04 Python 2.7.12
Installation and setup
There is a myriad of ways in which pyenv and pipenv could be installed. What follows is my own personal installation procedure for Linux systems.
Set up the pyenv root:
echo PYENV_ROOT=$HOME/Applications/python/pyenv > ~/.bashrc mkdir -p $PYENV_ROOT
~/.pam_environmentis a better although more complicated alternative.)
git clone https://github.com/pyenv/pyenv.git $PYENV_ROOT/
Add a convenience script:
(echo 'export PATH="$(dirname $(readlink -f $BASH_SOURCE))/bin:$PATH"' echo 'eval "$(pyenv init -)"') > $PYENV_ROOT/enable
Enable pyenv in the current shell:
Install a new Python version:
pyenv install 3.7.2
pip install pipenv