There are mainly two ways to install python packages in Databricks. This works on AWS & Azure Databricks.

  • Install Python package in the scope of notebook using pip magic command (%pip)
  • Install Python package in the cluster

Installation of python package in the notebook using pip magic command

The first option can be performed by any user who has Read and Write privilege in the notebook. The user does not need admin privileges for this installation. The package will get installed at the notebook level. So the packages will not be available to other users using the same cluster. We use the pip magic command to install the package (%pip).

The syntax to install package is given below. Please note that the % symbol is part of the command syntax. Run this command in the python notebook cell.

%pip install <package-name>
Example:  %pip install pyserial

Installation of python python package in the cluster

Admin or management privileges are required in this case. The package gets installed at the cluster level and the package will be available for all the other users using the same cluster. To perform this installation, select the cluster –> Click on the Libraries option –> click on install new. This option can be used to install Python packages, Java JAR files, R packages etc.

Select PyPI repository and enter the name of the python package. If you want to install any specific version, specify the version details also. For example, if you want to install the version 2.1 of Pyserial package, then specify the package as pyserial==2.1.

By default it downloads the package from the public PyPI repository. If you want to install it from a different repository, you can specify the repository URL in the Repository field.

Click on the install option to complete the installation. This will be installed at the cluster level and it will be available to all the users using the same cluster. After this reattach the notebooks to the cluster again to reflect the changes.

I hope this write up is useful. Feel free to comment if you have any questions or feedbacks.

Advertisement