Python

Several versions of Python, as well Miniconda and Anaconda which use the conda package manager, are available on Kamiak.

Using Python

There are several standalone versions of Python available on Kamiak. To use Python, simply load its module.

module load python3

or

module load python2

or

module load python # which loads python2

The Python modules available on Kamiak generally provide only the Python binaries, standard library, and pip. If you would like to use additional scientific Python libraries (SciPy, NumPy, etc.) you can instead use Anaconda.

Using Anaconda or Miniconda

Anaconda and Miniconda combine Python with the Conda package manager. Miniconda provides just Python and Conda. Anaconda provides many bundled packages in addition to Python and Conda.

With Conda, you can create and use environments that have different versions of Python and packages installed in them. Switching between environments is called activating the environment. Environments are a useful way to put together a set of packages with specific versions and dependencies. It is recommended to always use environments, which are local to your home directory in ./conda/envs. Otherwise, by default you will use the system-wide “root” environment, which does not allow you to modify or install packages.

We recommend using Python from within Anaconda or Miniconda, rather than running Python standalone.

To start using Anaconda, simply load its module.

module load anaconda3 # or miniconda3

From that point forward, you can load packages and manage environments using “conda” commands.

To see all available python packages on Kamiak, type:

conda list -n root

To see all created environments, type:

conda env list

Now let’s create a local environment called “myenv”.

conda create -n myenv

To activate the environment and use it, type:

source activate myenv

Now, let’s add the SciPy package from the Anaconda repository to our environment. We’ll specify a specific version, although this can be omitted to get the latest one.

conda install scipy=0.18.1

The package installs into the currently activated environment, but you can also specify an explicit environment name.

conda install -n myenv scipy=0.18.1

To use the activated environment, type:

python

To deactivate the environment, type:

source deactivate

Finally, to delete the environment, type:

conda remove -n myenv --all

Please see conda documentation for a very good description of managing environments and installing packages.

Conda commands are further described here.

Sharing Environments

If you wish to share an environment across a project’s users, you can append a --prefix env_path to the create and install commands to install packages into a central location you and others in your lab can use. The activate command similarly takes the full path instead of just a name. For example

conda create -n myenv --prefix /path-to-myenv
source activate /path-to-myenv/myenv

Installing Python Packages not Available on Kamiak

If you need to install Python packages that are not available in Kamiak’s installation of Anaconda, there are several ways to do this.

The recommended technique is to just use

conda install somePackage

If the package is not available in the Anaconda.org repository, you can specify another “channel” to search from, which are the locations where packages are stored:

conda install -n myenv -c someChannel somePackage

where someChannel is the identifier of the repository.

A second technique is to use pip from within Anaconda. Pip is another package manager that is fully integrated into Conda, and can be transparently used from within Conda environments. Pip is often used to install packages from respositories such as PyPI. The preferred sequence of commands to install a package using pip are as follows:

module load anaconda3
conda create -n myenv
source activate myenv
conda install pip
pip install somePackage

The above is the technique described in the conda documentation.
It is key to first install pip into your environment, since pip by default will try to install into wherever it is located, i.e., system directories, which you do not have permissions to modify.

A third technique, and not recommended, is to directly use pip from within standalone Python.
For example:

module load python3
pip install --user somePackage

The package will be installed into your home directory at ~/.local/lib/python3.7/site-packages, with any binaries in ~/.local/bin. After installation, you can run python3 and import the package. Note that the --user flag directs python to install the package off your home directory. If you would like your packages to be installed to a different directory use the --prefix flag. For example, to install packages into a central location, use the --prefix option combined with the PYTHONPATH variable.

pip3 install --prefix=~/myPython somePackage
export PYTHONPATH=~/myPython/lib/python3.7/site-packages:$PYTHONPATH

It is not recommended to use the above technique of pip within standalone Python, since this puts the package into a global environment in ~/.local that is searched by Python, outside the scope of managed environments. The better way to install is to use conda install pip, then install using pip. The package will then go into a specific conda environment rather than ~/.local.

See conda documentation for further information on installing packages in conda.

Installing from Source

To install a package from source, just follow the above directions, and instead of using a package name use the path to the somePackage.gzip or somePackage.tar.gz file.

For example, after activating your Conda environment and installing pip into it, type:

pip install /path/somePackage.tar.gz

Pip will automatically perform necessary extractions and run python with the setup.py file to complete the installation.