Note: If you’re pretty familiar with this issue, skip straight to the solution
This will be a quick post about how to solve an issue that’s been bugging me
the past couple of weeks since I upgraded to Ubuntu 20.04, which replaced my old
Python version with 3.8.
On my PC I have a folder full of Python projects I’ve created, going back years.
I try to keep dependencies isolated with virtualenvs, however one thing I
consistently forget to do is “freeze” the dependencies I use for each project
into a requirements.txt file.
I won’t go into a lot of detail on how virtualenvs work (see
this post
which covers how to use them), but essentially, the requirements file is a portable
list of your Python package dependencies and includes the version numbers used
in your project. When saved to a file, this is often checked in to source control
instead of the environment itself since the environment directory often contains
platform-specific binaries and things that may not work properly on someone else’s
PC (like links to the directory where you have installed Python).
In any case, when you’ve upgraded your Python version and “source” the environment,
you might not notice any issues at first. You get the right prefix in your shell
and your python command links to the environment Python, as expected:
(env) [email protected]:~/prg/py/elasticslurp$ which python
/home/user/prg/py/elasticslurp/env/bin/python
However, when you try to run a program that uses those packages, it doesn’t work 🙁
(env) [email protected]:~/prg/py/elasticslurp$ python main.py --help
Traceback (most recent call last):
File "main.py", line 2, in
from shodan import Shodan, APIError
ModuleNotFoundError: No module named 'shodan'
Oops. The problem is that each virtualenv is set up for a specific major and minor
version of Python (e.g. 3.6) and will not necessarily support the latest version
(e.g. 3.8). If you run into this issue, you won’t even be able to freeze the
dependencies because some critical files are missing:
(env) [email protected]:~/prg/py/elasticslurp$ pip freeze
Traceback (most recent call last):
File "/home/user/prg/py/elasticslurp/env/bin/pip", line 7, in
from pip import main
ModuleNotFoundError: No module named 'pip'
Let’s look at the make up of our environment directory and see if we can spot an
issue:
$ tree -L 2 env/
env/
├── bin
│ ├── activate
│ ├── activate.csh
│ ├── activate.fish
│ ├── bitmath
│ ├── chardetect
│ ├── easy_install
│ ├── easy_install-3.6
│ ├── f2py
│ ├── f2py3
│ ├── f2py3.6
│ ├── pip
│ ├── pip3
│ ├── pip3.6
│ ├── __pycache__
│ ├── python -> python3
│ ├── python3 -> /usr/bin/python3
│ ├── shodan
│ ├── vba_extract.py
│ └── wheel
├── include
├── lib
│ └── python3.6 # <-- suspicious
├── lib64 -> lib
├── pyvenv.cfg
└── share
└── python-wheels
There are certainly a lot of files labeled after the old version of Python. I’ve
called out one folder in particular, which is the environment’s site-packages
directory, containing a copy of all libraries you’ve installed. This is what
we need to change.
So fixing this issue enough to recover your installed libraries is pretty simple:
rename the old lib folder to match your own Python version, e.g.:
$ mv env/lib/python3.6/ env/lib/python3.8
Now freeze your dependencies to a file:
$ pip freeze > requirements.txt
Quickly check your requirements file and ensure it looks correct – I have not yet
run into problems with this, but I haven’t thoroughly tested edge cases. Next,
throw away the old environment and create a new one with your current Python version:
$ deactivate # this closes all shell references to the Python virtualenv
$ rm -r env/
$ python3 -m venv env
And finally, load your libraries:
$ source env/bin/activate
$ pip install -r requirements.txt
After this, your programs should work again.