Django Dependency Management with pip-compile and pip-tools
Even a basic Django project has multiple dependencies. After installing Django itself, various third-party packages--and their dependencies!--must be installed and managed. How do you track version numbers in a reproducible way?
In the JavaScript ecosystem, the npm package manager makes it easy to keep track of requirements. Every time a package is installed via npm install <package-name>, an additional file, package.json, is created that is automatically updated with the package name and version number.
Unfortunately, the Python package manager, pip, does not do that, so other approaches are required. In this post, we'll run through several options, including pip freeze and requirements.txt, and then demonstrate why pip-compile is a popular approach for many Django developers.
Django setup
Let's quickly go through the standard setup for a new Django project. Navigate to a new directory on the command line, create a fresh virtual environment, and install Django.
# Windows $ python -m venv .venv $ .venv\Scripts\Activate.ps1 (.venv) $ python -m pip install django~=4.2.0 # macOS $ python3 -m venv .venv $ source .venv/bin/activate (.venv) $ python3 -m pip install django~=4.2.0
Even at this introductory phase, we've installed Django and some of its dependencies.
pip freeze and requirements.txt
The command pip freeze lists all installed packages within a virtual environment and their version numbers. Let's try it now.
(.venv) $ pip freeze asgiref==3.7.2 Django==4.2.4 sqlparse==0.4.4
The latest version of Django, 4.2.4, relies on two other packages: asgiref and sqlparse.
You can output this to a physical file, typically called requirements.txt using the following command:
(.venv) $ pip freeze > requirements.txt
This file can be stored in source control using git. A fellow developer can download the code, create a fresh virtual environment, and run a command to install all the packages into their local environment.
(.venv) $ pip install -r requirements.txt
But this manual process requires the developer to regenerate a requirements.txt file every time a package is updated. It also doesn't directly address package dependencies.
pip-tools
A more sophisticated approach is using pip-tools, a command line tool for managing Python project dependencies. Let's install it and then run pip freeze to confirm it is installed correctly.
(.venv) $ python -m pip install pip-tools (.venv) $ pip freeze asgiref==3.7.2 build==0.10.0 click==8.1.7 Django==4.2.4 packaging==23.1 pip-tools==7.3.0 pyproject_hooks==1.0.0 sqlparse==0.4.4
Whoa, suddenly there are a lot more package dependencies to manage! Using our previous method, we would need to store these new packages manually in requirements.txt.
(.venv) $ pip freeze > requirements.txt
But at a high level, we only have two packages in our project: Django and pip-tools. All the other packages are dependencies of those two. With pip-tools, we can create a new root-level file called requirements.in to manage high-level dependencies. When compiled, it will automatically generate the full list of all dependencies into a requirements.txt file. Let's see this in action.
Since this tutorial is for demonstration purposes, use the deactivate command to exit the virtual environment. Delete the requirements.txt file using your text editor. Then delete the .venv directory. Note that you should always take great care when using the commands rm -rf together because you could inadvertently delete your entire computer if you're not careful. In this instance, we tell our computer to delete the entire .venv directory and its contents.
(.venv) $ deactivate $ rm -rf .venv
Now, create a fresh virtual environment and activate it. Then install pip-tools.
# Windows $ python -m venv .venv $ .venv\Scripts\Activate.ps1 (.venv) $ python -m pip install pip-tools # macOS $ python3 -m venv .venv $ source .venv/bin/activate (.venv) $ python -m pip install pip-tools
If you run pip freeze, you can see that pip-tools and all its dependencies are now installed in our virtual environment.
(.venv) $ pip freeze build==0.10.0 click==8.1.7 packaging==23.1 pip-tools==7.3.0 pyproject_hooks==1.0.0
Rather than use pip to install Django, we will add it to the requirements.in file. Here is the current project layout.
.venv/ ├── requirements.in
And we only need one line in the requirements.in file since we only install one high-level package.
# requirements.in django
To install Django and its dependencies into our virtual environment as well as a new requirements.txt file, run the following command:
(.venv) $ pip-compile requirements.in --strip-extras # # This file is autogenerated by pip-compile with Python 3.11 # by the following command: # # pip-compile --strip-extras requirements.in # asgiref==3.7.2 # via django django==4.2.4 # via -r requirements.in sqlparse==0.4.4 # via django
The command outputs the contents of the virtual environment and automatically adds it to the requirements.txt file for us.
# requirements.txt # # This file is autogenerated by pip-compile with Python 3.11 # by the following command: # # pip-compile --strip-extras requirements.in # asgiref==3.7.2 # via django django==4.2.4 # via -r requirements.in sqlparse==0.4.4 # via django
Add another package: environs
To see how pip-compile shines, try adding another package. Let's say it's environs, a way to implement environment variables that also comes with handy Django defaults.
Add it to the requirements.in file.
# requirements.in django environs[django]
Then, run the command pip-compile requirements.in --strip-extras to install it into the virtual environment.
(.venv) $ pip-compile requirements.in --strip-extras # # This file is autogenerated by pip-compile with Python 3.11 # by the following command: # # pip-compile --strip-extras requirements.in # asgiref==3.7.2 # via django dj-database-url==2.1.0 # via environs dj-email-url==1.0.6 # via environs django==4.2.4 # via # -r requirements.in # dj-database-url django-cache-url==3.4.4 # via environs environs==9.5.0 # via -r requirements.in marshmallow==3.20.1 # via environs packaging==23.1 # via marshmallow python-dotenv==1.0.0 # via environs sqlparse==0.4.4 # via django typing-extensions==4.7.1
The output is also now present in the requirements.txt file. And a good thing, too, because environs[django] has a bunch of dependencies: dj-database-url, dj-email-url, django-cache-url, marshmallow, and python-dotenv. Note that marshmallow has its own dependency, packaging, as does dj-database-url, which relies on typing-extensions.
# requirements.txt # # This file is autogenerated by pip-compile with Python 3.11 # by the following command: # # pip-compile --strip-extras requirements.in # asgiref==3.7.2 # via django dj-database-url==2.1.0 # via environs dj-email-url==1.0.6 # via environs django==4.2.4 # via # -r requirements.in # dj-database-url django-cache-url==3.4.4 # via environs environs==9.5.0 # via -r requirements.in marshmallow==3.20.1 # via environs packaging==23.1 # via marshmallow python-dotenv==1.0.0 # via environs sqlparse==0.4.4 # via django typing-extensions==4.7.1 # via dj-database-url
We've only installed two high-level packages at this point--Django and environs[django]--yet already managing our dependencies feels tricky. Thanks to pip-compile it doesn't have to be!
Updates
By default, pip-compile generates a requirements.txt file using the latest versions for the high-level packages in requirements.in. If an existing requirements.txt fulfills a dependency, no change will be made even if updates are available.
It is up to the developer to periodically update the packages. You can do this simultaneously with pip-compile --upgrade or update a specific package using --upgrade-package.
# only update the django package (.venv) $ pip-compile --upgrade-package django # update all packages (.venv) $ pip-compile --upgrade
See the pip-tools docs for complete instructions.