Python and Conda
Several versions of Python, as well Miniconda and Anaconda which use the conda package manager, are available on Kamiak.
Using Python
There are several standalone versions of Python available on Kamiak. To use Python, simply load its module.
module load python3
or
module load python2
or
module load python # which loads python2
The Python modules available on Kamiak generally provide only the Python binaries, standard library, and pip. If you would like to use additional scientific Python libraries (SciPy, NumPy, etc.) you can instead use Anaconda.
Using Anaconda or Miniconda
Anaconda and Miniconda combine Python with the Conda package manager. Miniconda provides just Python and Conda. Anaconda provides many bundled packages in addition to Python and Conda.
With Conda, you can create and use environments that have different versions of Python and packages installed in them. Switching between environments is called activating the environment. Environments are a useful way to put together a set of packages with specific versions and dependencies. It is recommended to always use environments, which are local to your home directory in ./conda/envs. Otherwise, by default you will use the system-wide “root” environment, which does not allow you to modify or install packages.
We recommend using Python from within Anaconda or Miniconda, rather than running Python standalone.
To start using Anaconda or Miniconda, simply load its module.
module load anaconda3 # or miniconda3
Do Not Use conda init
We do not recommend using conda init, it permanently alters your .bashrc so that the base environment is always activated in new shells. If installed, we recommend commenting out the conda init section from the end of your .bashrc.
Managing Environments
From that point forward, you can load packages and manage environments using “conda” commands.
Conda environments are grouped into locations or paths. Conda maintains a list of paths that are searched for environments available to you.
To view all your available environment paths, type:
conda config --show envs_dirs
To see all available python packages in the miniconda or anaconda installation, type:
conda list -n root
To see all available environments, type:
conda env list
Now let’s create a local environment called “myenv”.
conda create -n myenv
To activate the environment and use it, type:
source activate myenv
Now, let’s add the SciPy package from the Anaconda repository to our environment. We’ll specify a specific version, although this can be omitted to get the latest one.
conda install scipy=0.18.1
The package installs into the currently activated environment, but you can also specify an explicit environment name.
conda install -n myenv scipy=0.18.1
To use the activated environment, type:
python
To deactivate the environment, type:
source deactivate
Finally, to delete the environment, type:
conda remove -n myenv --all
Please see conda documentation for a description of managing environments and installing packages.
Conda commands are further described here.
Sharing Environments
If you wish to share an environment across a lab’s users, you can append a --prefix env_path to the create and install commands to install packages into a central location you and others in your lab can use. The activate command similarly takes the full path instead of just a name. For example
conda create -n myenv --prefix /path-to-envs source activate /path-to-envs/myenv
As mentioned previously, conda environments are grouped into locations or paths. The list of paths that are searched for environments available to you is in the bash environment variable CONDA_ENV_PATH. It can also be set in a condarc file as follows:
envs_dirs:
- /data/lab/mylab/conda/envs # appends to existing search path
env_prompt: ({name})
So instead of always providing an absolute path in your source activate commands, you can set up a condarc file as shown above,
for example in:
/data/lab/mylab/conda/startup/condarc
and then have it invoked automatically to set the environment paths for your lab by editing your bashrc (or a lab modulefile):
export CONDARC=/data/lab/mylab/conda/startup/condarc
Then, to invoke one of your lab’s environments, you can simply do:
source activate my-shared-env
Installing Packages Locally using Conda
If you need to install Python packages that are not otherwise available in Kamiak’s installation, there are several ways to do this.
The recommended technique is to just use
conda install somePackage
If the package is not available in the Anaconda.org repository, you can specify another “channel” to search from, which are the locations where packages are stored:
conda install -n myenv -c someChannel somePackage
where someChannel is the identifier of the repository.
A second technique is to use pip from within Anaconda. Pip is another package manager that is fully integrated into Conda, and can be transparently used from within Conda environments. Pip is often used to install packages from respositories such as PyPI. The preferred sequence of commands to install a package using pip are as follows:
module load anaconda3 conda create -n myenv source activate myenv conda install pip pip install somePackage
The above is the technique described in the conda documentation.
It is key to first install pip into your environment, since pip by default will try to install into wherever it is located, i.e., system directories, which you do not have permissions to modify.
A third technique, and not recommended, is to directly use pip from within standalone Python.
For example:
module load python3 pip install --user somePackage
The package will be installed into your home directory at ~/.local/lib/python3.7/site-packages, with any binaries in ~/.local/bin. After installation, you can run python3 and import the package. Note that the --user flag directs python to install the package off your home directory. If you would like your packages to be installed to a different directory use the --prefix flag. For example, to install packages into a central location, use the --prefix option combined with the PYTHONPATH variable.
pip3 install --prefix=~/myPython somePackage export PYTHONPATH=~/myPython/lib/python3.7/site-packages:$PYTHONPATH
It is not recommended to use the above technique of pip within standalone Python, since this puts the package into a global environment in ~/.local that is searched by Python, outside the scope of managed environments. The better way to install is to use conda install pip, then install using pip. The package will then go into a specific conda environment rather than ~/.local.
See conda documentation for further information on installing packages in conda.
Installing from Source
To install a package from source, just follow the above directions, and instead of using a package name use the path to the somePackage.gzip or somePackage.tar.gz file.
For example, after activating your Conda environment and installing pip into it, type:
pip install /path/somePackage.tar.gz
Pip will automatically perform necessary extractions and run python with the setup.py file to complete the installation.