Installing Python packages in 2019: pyenv and pipenv

Gioele Barabucci,

The way Python packages are installed and managed used to be quite convoluted with multiple conflicting alternatives coming and going every few months.

As of 2019, thanks to the efforts of many groups including the Python Packaging Authority (PyPa), the situation is now much better and two clear winners are emerging: pyenv and pipenv. Similar names, completely different tasks.

(<ruby-appreciation-moment>Both projects are heavily influenced by their Ruby counterparts: rbenv and bundler.</ruby-appreciation-moment>)

The objective of this small guide is to describe how pyenv, pipenv and various other tools work together to install and manage Python packages. This guide is aimed at those who would like to have a look behind the scenes to understand how modern Python packaging tools work together.

The problem

In your Python program you use foolib, a library that is not installed by default. Not a big problem, just install it via pip install foolib, right?

Not so easy. Here is a couple of issue with that.

  1. Do you have enough rights to install that package? Maybe you need to use sudo pip. This sounds wrong.

  2. Your application needs version 2.0 of foolib. Another Python application in you system needs version 1.8. Oops, API conflict.

  3. Other people that will install your application will need foolib. You need to document it somewhere, for example in requirements.txt. But using pip freeze will record the exact version of foolib you happen to use right now. Any version of the 2.x series would be OK, but there is no easy way to tell this to pip.

  4. Actually all this is moot because you need native datatypes and they are available only in Python 3.7. Unfortunately your Linux distro only ships version 3.5.

All these problems could be solved in many different ways.

Obligatory XKCD: 1987: Python Environment.

As of 2019, the state of the art in installing packages and managing dependencies in Python applications consists in using pyenv and pipenv.

In the following sections we will take a step back and have a look at the tasks carried out by pyenv and pipenv, as well as peeking under the hood to understand their relation to other tools like the good ol' pip and virtualenv.

The pieces of the puzzle

If you just want to install and use Python packages, there are only two tools that you should care about and use directly: pyenv and pipenv. However, to better understand how things work, you should also know a bit about two other tools used internally by pipenv: pip and virtualenv.

Here is a small description of what each of these tools does.

pyenv
allows you to install and use specific versions of Python and its related tools.

In particular, pyenv allows you to install many different versions of Python on the same system and by non-root users. Specific versions can then be chosen at runtime. pyenv plays with environment variables ($PATH) and symlinks to provide you with a specific version of the python executable.

For example, you can run pyenv shell 3.7.2 and, from that point on, whenever you run python, the executable for version 3.7.2, installed under $PYENV_ROOT, will be used instead of the whatever other version of Python is installed in your system.

pip
fetches and installs Python packages.

More precisely, it installs packages into an existing Python installation, for example in /usr/lib/python3/dist-packages/. If pip is run inside a pyenv environment, it will install the packages into the currently enabled Python environment, somewhere under $PYENV_ROOT.

virtualenv
tricks pip into installing packages into arbitrary directories such as ~/.virtualenvs/foobar/.

virtualenv allows you to keep the Python packages of different projects isolated from one another. This is helpful, for example, if different projects require different incompatible versions of the same package.

pipenv
is the thing that puts all these tools together.

It allows you to specify in a project-specific Pipenv file the direct dependencies of your project as well as the required Python version.

pipenv has two main phases or modes: installation mode and runtime mode.

During the installation phase (when you run pipenv install), pipenv takes care of:

  • finding the best set of dependencies that fulfils the project's requirements in terms of versions and compatibilities.
  • using virtualenv to create a project-specific package directory where the dependencies of the project can be installed.
  • calling pip to actually install these dependencies.

At runtime (when you run pipenv shell or pipenv run COMMAND), pipenv takes care of:

  • using pyenv to create a runtime environment with the specified version of Python.
  • integrate the installed dependencies into the current environment.

The big picture

To recap:

pyenv
takes care of the python binary and all related tools. It stores everything under $PYENV_ROOT.
pipenv
takes care of
  1. calculating the complete set of dependencies;
  2. (in installation mode) telling virtualenv and pip where to install the dependencies;
  3. (in runtime mode) making available an environment with the right version of the Python interpreter (via pyenv) and the right set of packages (via virtualenv).

The only commands you should care about are:

An example in practice

Let's see how things work together with a small example.

We are developing a Python application in ~/Projects/exapp. We already installed and set up pyenv and pipenv (discussed in the following section).

This is the code of our wonderful application:

#!/usr/bin/env python

import crayons
from pyfiglet import figlet_format

print(crayons.blue(figlet_format('Hi there!'), bold=True))

As you can see, we use a couple of non-standard modules. We declare these dependencies in the Pipfile:

[[source]]
url = "https://pypi.python.org/simple"

[packages]
crayons = "*"
pyfiglet = "*"

[requires]
python_version = "3.7.2"

At this point we run pipenv install to install all the needed packages. The output of pipenv install will show us how the various pieces of puzzle fit together.

(Note: The output is slightly redacted for the sake of brevity.)

Creating a virtualenv for this project…
Pipfile: $HOME/Projects/exapp/Pipfile
Using $PYENV_HOME/shims/python (3.7.2) to create virtualenv…
Running virtualenv with interpreter $PYENV_HOME/shims/python
Using base prefix '$PYENV_HOME/versions/3.7.2'
New python executable in $HOME/.virtualenvs/exapp-nXCyFRyd/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: $HOME/.virtualenvs/exapp-nXCyFRyd
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (a0242d)!
Installing dependencies from Pipfile.lock (a0242d)…
Installing 'colorama'▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/3 — 00:00:00
$ ['$HOME/.virtualenvs/exapp-nXCyFRyd/bin/pip', 'install',
'--verbose', '--upgrade', '--no-deps', '-r',
'"/tmp/pipenv-hhv176qm-requirements/pipenv-9zs0b2h7-requirement.txt"',
'-i', 'https://pypi.python.org/simple', '--require-hashes']
Installing 'pyfiglet'▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/3 — 00:00:00
$ ['$HOME/.virtualenvs/exapp-nXCyFRyd/bin/pip', 'install',
'--verbose', '--upgrade', '--no-deps', '-r',
'"/tmp/pipenv-hhv176qm-requirements/pipenv-zaq1e563-requirement.txt"',
'-i', 'https://pypi.python.org/simple', '--require-hashes']
Installing 'crayons'▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/3 — 00:00:00
$ ['$HOME/.virtualenvs/exapp-nXCyFRyd/bin/pip', 'install',
'--verbose', '--upgrade', '--no-deps', '-r',
'"/tmp/pipenv-hhv176qm-requirements/pipenv-n9xxkvj7-requirement.txt"',
'-i', 'https://pypi.python.org/simple', '--require-hashes']
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 3/3 — 00:00:02
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

Here is a breakdown of the main steps performed by pipenv:

  1. pyenv is used to select the version of Python specified in the Pipfile (version 3.7.2).
  2. A virtual environment specific to our project exapp-nXCyFRy is created. (The name is created by hashing the absolute path of the Pipenv file.)
  3. The dependency graph is first calculated and then recorded in the Pipfile.lock file (Pipfile.lock is a story for another time).
  4. pip (the one linked in the virtualenv and provided by pyenv) is used to install the dependencies (including the dependencies of the dependencies), one by one.

At this point the environment where our application could run is ready, but it is not active yet.

We have two options to run our application in the prepared environment: launching a single-shot command or opening an interactive shell.

The most direct way is to run the application as a one-time command. Pipenv will load the environment for us, run the application and then exit the environment. We can do this using pipenv run ./exapp.py.

$ pipenv run ./exapp.py
 _   _ _   _   _                   _
| | | (_) | |_| |__   ___ _ __ ___| |
| |_| | | | __| '_ \ / _ \ '__/ _ \ |
|  _  | | | |_| | | |  __/ | |  __/_|
|_| |_|_|  \__|_| |_|\___|_|  \___(_)

$

The second way is to open a shell inside the environment from which we can launch our application or execute any other command. To do this we use pipenv shell. For example:

$ pipenv shell
Launching subshell in virtual environment…
 . $HOME/.virtualenvs/exapp-nXCyFRyd/bin/activate
$ ./exapp.py
 _   _ _   _   _                   _
| | | (_) | |_| |__   ___ _ __ ___| |
| |_| | | | __| '_ \ / _ \ '__/ _ \ |
|  _  | | | |_| | | |  __/ | |  __/_|
|_| |_|_|  \__|_| |_|\___|_|  \___(_)

$ python --version # inside the environment
Python 3.7.2
$ exit
$ python --version # outside, in Ubuntu 16.04
Python 2.7.12

Installation and setup

There is a myriad of ways in which pyenv and pipenv could be installed. What follows is my own personal installation procedure for Linux systems.

  1. Set up the pyenv root:

    echo PYENV_ROOT=$HOME/Applications/python/pyenv > ~/.bashrc
mkdir -p $PYENV_ROOT

    (Setting PYENV_ROOT in ~/.pam_environment is a better although more complicated alternative.)

  2. Checkout pyenv:

    git clone https://github.com/pyenv/pyenv.git $PYENV_ROOT/

  3. Add a convenience script:

    (echo 'export PATH="$(dirname $(readlink -f $BASH_SOURCE))/bin:$PATH"'
echo 'eval "$(pyenv init -)"') > $PYENV_ROOT/enable

  4. Enable pyenv in the current shell:

    . ~/Applications/python/pyenv/enable

  5. Install a new Python version:

    pyenv install 3.7.2

  6. Install pipenv:

    pip install pipenv