Advertisements

How to maintain packages in a python project ?

In most of the cases, we might need external packages for the development of a python program. These external packages are either available in pypi repository or available locally as archive files. Usually people just installs the packages directly in the python environment using pip command.

The pip command by default installs the package from the pypi repository. If we are not specifying the version, it selects the latest available version of that package supported by the python present in the environment (Python 2 or 3). Because of this nature, the pip command will always pick up the latest version of the packages. The packages may undergo drastic changes in newer releases. For example, an application developed with version X of a package may not work with the version Y of the same package. So simply noting down the package names itself will not help to manage the project. We need the list of all packages with the versions. Also manually installing the packages one by one is also a difficult task, because there can be several tens of packages within a single project.

The best practices for managing packages in a project are

  1. Use python Virtual Environment.
  2. Create a requirements.txt to maintain the package details.

The details on how to create and use virtual environment is explained in my previous post.

requirements.txt is a simple text file to maintain all the package dependencies with versions. A sample format is given below

Click==7.0
Flask==1.0.2
Flask-Cors==3.0.7
itsdangerous==1.1.0
Jinja2==2.10
MarkupSafe==1.1.0
six==1.12.0
Werkzeug==0.14.1

Packages can be installed using a single command

pip install -r requirements.txt

Packages in an environment can be captured in a requirements.txt file in one shot using the following command.

pip freeze > requirements.txt

This practice will help developers to manage the dependency list and easy code migration.

Advertisements

Virtual environment in Python

What is a Virtual Environment ?

A virtual environment is a tool that helps developers to segregate and maintain the dependencies required by different projects by creating isolated python virtual environments.

Need for Virtual Environment ?

Suppose,  User A and User B are working on 2 projects.

The package requirements for user A is given below

Click==7.0
Flask==1.0.2
Flask-Cors==3.0.7
itsdangerous==1.1.0
Jinja2==2.10
MarkupSafe==1.1.0
six==1.12.0
Werkzeug==0.14.1

Also the package requirements for user B is given below.

Click==6.0
Flask==1.0.1
Flask-Cors==3.0.2
itsdangerous==1.1.0
Jinja2==2.10
MarkupSafe==1.1.0
six==1.11.0
Werkzeug==0.11.2

As you can see, the two developers use different versions of similar packages. Python does not have the ability to differentiate between multiple versions of the same package in the site-packages directory.  By default the package installation happens in the default site-packages directory of the python installation. In Unix like operating system, by default the location will be owned by the root user and a normal user will not be able to perform a package installation without elevated privileges.

Virtual environment plays its role in the following scenarios

  • To isolate the dependencies required for different projects
  • To maintain the base python packages untouched. In case of multi-user environment, upgrading or modifying a package might disrupt the operation of one or more projects.
  • To enable easy access for the installation and management of python packages to the end users without enabling system level elevated privileges.
  • To easily manage the dependencies used in a specific project. We can copy the virtual environment to another system of the same version. Also it is easy to replicate the environment by dumping the package list (pip freeze).

How to create a virtual environment ?

For creating a virtual environment, we need the virtual environment package installed in the base python environment.

As root user or with elevated privilege, execute the following command

pip install virtualenv

Then create the virtual environment with the following command. The virtualenv command will be available only if the package was installed in the base python.

virtualenv <path for virtual environment>

You can specify any writable location as the path for virtual environment. All the virtual environment related files and packages will be installed in this directory. It may take few seconds complete the virtual environment setup.

Now type

which python

this will be still pointing to the base python. For using the virtual environment, we need to activate the environment.

source <path of virtual environment>/bin/activate

The above command will activate the virtual environment in the current session. For making it enabled in all sessions by default, add these lines in the .bashrc file.

Now again type which python check the result. It will be pointing to the newly created virtual environment.

For deactivating the environment, simply type deactivate in the command line.