Python
Several versions of Python, as well Miniconda and Anaconda which use the conda package manager, are available on Kamiak.
Using Python
There are several standalone versions of Python available on Kamiak. To use Python, simply load its module.
module load python3
or
module load python2
or
module load python # which loads python2
The Python modules available on Kamiak generally provide only the Python binaries, standard library, and pip. If you would like to use additional scientific Python libraries (SciPy, NumPy, etc.) you can instead use Anaconda.
Using Anaconda or Miniconda
Anaconda and Miniconda combine Python with the Conda package manager. Miniconda provides just Python and Conda. Anaconda provides many bundled packages in addition to Python and Conda.
With Conda, you can create and use environments that have different versions of Python and packages installed in them. Switching between environments is called activating the environment. Environments are a useful way to put together a set of packages with specific versions and dependencies. It is recommended to always use environments, which are local to your home directory in ./conda/envs. Otherwise, by default you will use the system-wide “root” environment, which does not allow you to modify or install packages.
We recommend using Python from within Anaconda or Miniconda, rather than running Python standalone.
To start using Anaconda, simply load its module.
module load anaconda3 # or miniconda3
From that point forward, you can load packages and manage environments using “conda” commands.
To see all available python packages on Kamiak, type:
conda list -n root
To see all created environments, type:
conda env list
Now let’s create a local environment called “myenv”.
conda create -n myenv
To activate the environment and use it, type:
source activate myenv
Now, let’s add the SciPy package from the Anaconda repository to our environment. We’ll specify a specific version, although this can be omitted to get the latest one.
conda install scipy=0.18.1
The package installs into the currently activated environment, but you can also specify an explicit environment name.
conda install -n myenv scipy=0.18.1
To use the activated environment, type:
python
To deactivate the environment, type:
source deactivate
Finally, to delete the environment, type:
conda remove -n myenv --all
Please see conda documentation for a very good description of managing environments and installing packages.
Conda commands are further described here.
Sharing Environments
If you wish to share an environment across a project’s users, you can append a --prefix env_path to the create and install commands to install packages into a central location you and others in your lab can use. The activate command similarly takes the full path instead of just a name. For example
conda create -n myenv --prefix /path-to-myenv source activate /path-to-myenv/myenv
Installing Python Packages not Available on Kamiak
If you need to install Python packages that are not available in Kamiak’s installation of Anaconda, there are several ways to do this.
The recommended technique is to just use
conda install somePackage
If the package is not available in the Anaconda.org repository, you can specify another “channel” to search from, which are the locations where packages are stored:
conda install -n myenv -c someChannel somePackage
where someChannel is the identifier of the repository.
A second technique is to use pip from within Anaconda. Pip is another package manager that is fully integrated into Conda, and can be transparently used from within Conda environments. Pip is often used to install packages from respositories such as PyPI. The preferred sequence of commands to install a package using pip are as follows:
module load anaconda3 conda create -n myenv source activate myenv conda install pip pip install somePackage
The above is the technique described in the conda documentation.
It is key to first install pip into your environment, since pip by default will try to install into wherever it is located, i.e., system directories, which you do not have permissions to modify.
A third technique, and not recommended, is to directly use pip from within standalone Python.
For example:
module load python3 pip install --user somePackage
The package will be installed into your home directory at ~/.local/lib/python3.7/site-packages, with any binaries in ~/.local/bin. After installation, you can run python3 and import the package. Note that the --user flag directs python to install the package off your home directory. If you would like your packages to be installed to a different directory use the --prefix flag. For example, to install packages into a central location, use the --prefix option combined with the PYTHONPATH variable.
pip3 install --prefix=~/myPython somePackage export PYTHONPATH=~/myPython/lib/python3.7/site-packages:$PYTHONPATH
It is not recommended to use the above technique of pip within standalone Python, since this puts the package into a global environment in ~/.local that is searched by Python, outside the scope of managed environments. The better way to install is to use conda install pip, then install using pip. The package will then go into a specific conda environment rather than ~/.local.
See conda documentation for further information on installing packages in conda.
Installing from Source
To install a package from source, just follow the above directions, and instead of using a package name use the path to the somePackage.gzip or somePackage.tar.gz file.
For example, after activating your Conda environment and installing pip into it, type:
pip install /path/somePackage.tar.gz
Pip will automatically perform necessary extractions and run python with the setup.py file to complete the installation.