Virtual Environment and Packages

Sometimes there is a need to create virtual environment, a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages as there is sometimes a need for specific version of python library. Here come venv

Creating Virtual Environments

% module load python/3/intel/2020
% python3 -m venv myenv

This will create a myenv directory if it doesn’t exist, and also create directories inside it containing a copy of the Python interpreter, the standard library, and various supporting files.

% source myenv/bin/activate

Activating the virtual environment will change your shell’s prompt to show what virtual environment you’re using,

(tutorial-env) [user1@hpc-node1 bin]$ python
Python 3.7.4 (default, Nov 22 2019, 21:31:39)
[GCC 7.3.0] :: Intel(R) Corporation on linux
Type "help", "copyright", "credits" or "license" for more information.
Intel(R) Distribution for Python is brought to you by Intel Corporation.
Please check out: https://software.intel.com/en-us/python-distibution
>>> import sys
>>> sys.path
['', '/usr/local/intelpython3/lib/python37.zip', ....., '.....', '/myhome/melvin/python_project/tutorial-env/lib/python3.7/site-packages']
>>>

Managing Packages with pip

You can install, upgrade and remove using a program called pip. By default, pip will install packages from the Python Package Index (http://pypi.org).

You can install the latest version of a package by specifying a package’s name:

(tutorial-env) [user1@hpc-node1 bin]$ python -m pip install novas
Collecting novas
  Downloading https://files.pythonhosted.org/packages/42/95/a05bc35cb119925e10f9faa8a2bd17020b0a585744a38921a709acdd9a14/novas-3.1.1.5.tar.gz (135kB)
    100% |████████████████████████████████| 143kB 4.6MB/s
Installing collected packages: novas
  Running setup.py install for novas ... done
Successfully installed novas-3.1.1.5

You can also install specific version by using the command

(tutorial-env) [user1@hpc-node1 bin]$ python -m pip install requests==2.6.0
Collecting requests==2.6.0
  Downloading https://files.pythonhosted.org/packages/73/63/b0729be549494a3e31316437053bc4e0a8bb71a07a6ee6059434b8f1cd5f/requests-2.6.0-py2.py3-none-any.whl (469kB)
    100% |████████████████████████████████| 471kB 4.3MB/s
Installing collected packages: requests
Successfully installed requests-2.6.0

You can also upgrade the package by using the command

(tutorial-env) [user1@hpc-node1 bin]$ python -m pip install --upgrade request    s
Collecting requests
  Downloading https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50    280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl     (61kB)
    100% |████████████████████████████████| 61kB 3.4MB/s
Collecting chardet<5,>=3.0.2 (from requests)
  Downloading https://files.pythonhosted.org/packages/19/c7/fa589626997dd07bd87    d9269342ccb74b1720384a4d739a1872bd84fbe68/chardet-4.0.0-py2.py3-none-any.whl (1    78kB)
    100% |████████████████████████████████| 184kB 5.3MB/s
Collecting urllib3<1.27,>=1.21.1 (from requests)
  Downloading https://files.pythonhosted.org/packages/0c/cd/1e2ec680ec7b09846dc    6e605f5a7709dfb9d7128e51a026e7154e18a234e/urllib3-1.26.5-py2.py3-none-any.whl (    138kB)
    100% |████████████████████████████████| 143kB 5.9MB/s
Collecting idna<3,>=2.5 (from requests)
  Downloading https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6    f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 4.3MB/s
Collecting certifi>=2017.4.17 (from requests)
  Downloading https://files.pythonhosted.org/packages/05/1b/0a0dece0e8aa492a6ec    9e4ad2fe366b511558cdc73fd3abc82ba7348e875/certifi-2021.5.30-py2.py3-none-any.wh    l (145kB)
    100% |████████████████████████████████| 153kB 5.1MB/s
Installing collected packages: chardet, urllib3, idna, certifi, requests
  Found existing installation: requests 2.6.0
    Uninstalling requests-2.6.0:
      Successfully uninstalled requests-2.6.0
Successfully installed certifi-2021.5.30 chardet-4.0.0 idna-2.10 requests-2.25.    1 urllib3-1.26.5

You can display all the packages in the environment in virtual environment

(tutorial-env) [user1@hpc-node1 bin]$ pip list
Package    Version
---------- ---------
certifi    2021.5.30
chardet    4.0.0
idna       2.10
novas      3.1.1.5
pip        19.0.3
requests   2.25.1
setuptools 40.8.0
urllib3    1.26.5

pip freeze can be used to produce a similar list of installed packages and formatted in a output file that pip install can read from

(tutorial-env) [user1@hpc-node1 bin]$ pip freeze > requirements.txt
(tutorial-env) [user1@hpc-node1 bin]$ cat requirements.txt
certifi==2021.5.30
chardet==4.0.0
idna==2.10
novas==3.1.1.5
requests==2.25.1
urllib3==1.26.5

Install all the necessary packages with install -r from requirements.txt

(tutorial-env) [user1@hpc-node1 bin]$ python -m pip install -r requirements.txt
Requirement already satisfied: certifi==2021.5.30 in /myhome/melvin/python_project/tutorial-env/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (2021.5.30)
Requirement already satisfied: chardet==4.0.0 in /myhome/melvin/python_project/tutorial-env/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (4.0.0)
Requirement already satisfied: idna==2.10 in /myhome/melvin/python_project/tutorial-env/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (2.10)
Requirement already satisfied: novas==3.1.1.5 in /myhome/melvin/python_project/tutorial-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (3.1.1.5)
Requirement already satisfied: requests==2.25.1 in /myhome/melvin/python_project/tutorial-env/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (2.25.1)
Requirement already satisfied: urllib3==1.26.5 in /myhome/melvin/python_project/tutorial-env/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.26.5)

References:

• Virtual Environment and Packages (docs.python.org)