Storage and File Permissions
Kamiak provides several types of data storage that is available for users and lab groups.
|Name||Mount on Kamiak||Snapshots for Data Recovery?||ACL Support?||Description|
|Home||/home||Yes||Yes||Storage provided to all users, 100GB per user (effective June 2021).|
|Lab||/data/lab||Yes||Yes||Storage provided to all PI's, 500GB per PI (effective June 2021). Additional project storage is available for annual rental in 500GB units.|
|Scratch||/scratch||No||No||Scratch storage, with 2 week folder lifetime, limited to 10TB per user.|
|Local Scratch||/local||No||No||Scratch storage on each compute node, with 2 week folder lifetime.|
All storage other than local scratch is a shared resource that can be accessed on every compute node. If you create a file in your home directory on a login node, that same file is also available within a job on compute nodes. As such, there’s usually no need to copy files between compute nodes.
Storage on Kamiak is designed to be efficient, performant, and reliable. However, there are NO backups of any data residing on Kamiak. Data owners (you) are responsible for protecting their own data by copying it out of Kamiak onto another system.
All Kamiak users are provided with a home directory. Home directories are named using your WSU NID and your usage can be viewed using the
$ echo $HOME /home/my.NID $ cd $HOME $ quota -s -f /home Disk quotas for user my.NID (uid 8003): Filesystem space quota limit grace files quota limit grace 10.110.0.11:/home 8001M 10240M 10240M 139k 4295m 4295m
The size of your home directories cannot be increased. To acquire more space you will need to use other storage, such as Project space or Scratch space.
Each PI’s lab group also gets a folder in /data/lab named after their lab group, with a default size specified in the table at the top of this page. Additional project storage is available for rent on an annual basis through the CIRC service center.
For more information about renting additional project storage, see the Become an Investor page.
The cluster also provides temporary scratch storage that can be used without cost by all users. The following guidelines apply on using it:
- To use scratch, you must create a folder (called a “workspace”) to keep your data in. You must use “mkworkspace” as described below to create the scratch folder, otherwise it may be subject to deletion.
- Each workspace and all data within it has a maximum lifetime of 2 weeks, after which it is removed automatically.
- Each user is limited to 10TB of scratch space. User quotas are in effect, and will warn you if you exceed the limit.
- This is a shared resource with finite space. Care should be taken to ensure appropriate use of it.
There are two types of scratch storage available.
- Scratch. The first is scratch storage available on the cluster’s shared file system.
- Local. The second is a local SSD flash drive within each compute node. Since this storage is only accessible by that single compute node, and the disk is small (about 400GB), its use is generally not recommended.
Creating and listing scratch folders
To use scratch storage, you’ll need to create a “workspace” folder in which to store your data. The maximum lifetime of a scratch workspace is 2 weeks. Scratch storage is not intended for permanent data and users should consider project space for that use.
Let’s look at an example of creating and listing a workspace. Run each command with the option
--help to see what options are available.
mkworkspace to create a scratch folder. Run the command either on a login node, within an interactive idev session on a compute node, or within a batch job.
$ myscratch="$(mkworkspace)" $ echo $myscratch /scratch/user/my.netid/20210422_170154
You can also specify a prefix to the system-generated scratch folder name:
$ myscratch="$(mkworkspace -n myprefix)" $ echo $myscratch /scratch/user/my.netid/myprefix_20210422_170154
For example, a prefix can be used to prepend the job number to a scratch folder created inside a job:
$ myscratch="$(mkworkspace -n $SLURM_JOBID)" $ echo $myscratch /scratch/user/my.netid/31743729_20210422_170154
where 31743729 is the job number. The other fields are the date and time.
If you need to use a scratch folder across jobs, but don’t want to change your job scripts every time the scratch folder name changes, you can either create a symbolic link to the generated unique name or set an environment variable, and use that in your job scripts, although remember that the link or variable has to be changed when the scratch folder expires.
lsworkspace to list all your workspaces and their expiration dates.
$ lsworkspace /scratch/user/my.netid/20210422_170154 Created: May 02 2020 Expires: May 16 2020
To create and list local workspaces (only supported on compute nodes):
[cn32]$ myscratch="$(mkworkspace -l)" [cn32]$ echo $myscratch /local/scratch/user/my.netid/20210422_170154 [cn32]$ lsworkspace -l /local/scratch/user/my.netid/20210422_170154 Created: May 02 2020 Expires: May 16 2020
rm to delete the contents of a scratch folder.
rm -r -I /scratch/user/my.netid/20210422_170154/*
While you can delete the contents within a scratch folder, you cannot delete the scratch folder itself.
Shortly after a workspace expires, the folder and all of the contents within it will be deleted.
Batch job script example
Now let’s use a workspace in a batch job script:
#!/bin/bash #SBATCH --time=0-00:010:00 ### Wall clock time limit in Days-HH:MM:SS #SBATCH --ntasks-per-node=1 ### Number of tasks to be launched per Node myscratch="$(mkworkspace)" echo "My workspace is: $myscratch" cd "$myscratch" echo 'Hello!' > file.txt
This job will simply create a workspace, place the text “Hello!” into a file in the workspace, and end the job.
It is good practice to remove workspace contents as soon as they are no longer needed rather than allowing them to expire. This frees up the storage space for others to use.
Although data on Kamiak is not backed up, storage snapshots are created periodically on both Home and Project storage, but not on other storage including Scratch and Local Scratch. These snapshots are read-only and can be used to recover data that has been lost or over-written recently.
When you delete or modify files, for home and project folders, the original data is preserved for 3 days in snapshots. You can see your snapshots in a “.snapshot” subfolder within any directory, ordered by date. Snapshots count towards total disk utilization (df), while actual user data is reflected in (du). So it is possible that even after deleting files you could run out of space. If space is a problem, you can wait 3 days for the changes to rotate off, or you can submit a service request to have the system administrators delete snapshots or change the snapshot policy to 0-2 days, if that helps. However, snapshots can only be deleted or have their policies changed at the volume level, e.g. /data/somePI/.snapshot/daily.2019-09-16_0010, not for specific subfolders or files.
It is imperative that when data loss occurs users act quickly to recover it from a snapshot. Otherwise, the data will be lost permanently when the snapshots expire and are automatically removed.
Snapshots are contained in a hidden folder called .snapshot. Note the leading dot and the fact that this directory is invisible (even with the -a option of
ls) but its contents can be viewed directly:
$ ls -1 /home/.snapshot daily.2017-01-03_0010 daily.2017-01-04_0010 daily.2017-01-05_0010
$ ls -1 /data/myLabName/.snapshot daily.2017-01-03_0010 daily.2017-01-04_0010 daily.2017-01-05_0010
To recover data from a snapshot simply identify which snapshot contains the data then copy it out of the snapshot and into your storage:
$ rm -f test.py # oops $ ls test.py ls: cannot access test.py: No such file or directory $ cp /home/.snapshot/hourly.2017-01-05_1005/myWSU.NID/test.py test.py $ ls test.py test.py
In the above example “hourly.2017-01-05_1005” is the most recent snapshot, so that’s where we copy our data from. Any changes made to test.py since this snapshot was made are lost and unrecoverable. It is important to act quickly and copy the data out of the snapshot before it expires and the data is lost forever.
The time, frequency, and retention of snapshots vary between storage spaces on Kamiak. Owners of Project space can request a custom snapshot schedule or disable them entirely. The latter may be desirable if the data has a high rate of change which causes snapshots to become large and consume capacity excessively. Submit a Service Request to learn more or request a change.
Storage on Kamiak uses Linux file ownership and permissions for data security. An explanation of how ownership and permissions behave in Linux can be found at this site and in various other guides available online.
As data owners on Kamiak, you are responsible for the security of their data and must allow or restrict access to their data as needed. Importantly, allowing “other” the write permission (i.e.
chmod 777) is considered dangerous. You are highly encouraged to utilize file permission groups to allow specific additional users or groups to access your data.
Each file and directory has three sets of permissions for it: one for the owning user (u), one for the owning group (g), and one for everyone else (o). Each permission is one or more of read (r), write (w), and execute file/traverse folder (x). Each user belongs to a primary group, which is used as the group owner when creating files, and also belongs to one or more supplemental groups, which can also be used to access files. You can see the groups your are in using
Let’s look at an example:
$ cd /data/myLabName/ $ mkdir example $ ls -ld example/ drwxr-xr-x 2 my.NID its_p_sys_ur_kam-its 4096 Aug 10 08:14 example
In that example we created a new directory in project space. The file is owned by my user and my lab group. However, the permissions
rwxr-xr-x show that group only has read access. Let’s allow our group to write data into the directory:
$ chmod g+w example/ $ ls -ld example/ drwxrwxr-x 2 my.NID its_p_sys_ur_kam-its 4096 Aug 10 08:14 example/
We can also secure the directory by preventing anyone else (users not in our lab group) from accessing data in the directory:
$ chmod o-rwx example/ $ ls -ld example/ drwxrwx--- 2 my.NID its_p_sys_ur_kam-its 4096 Aug 10 08:14 example/
By default, files and directories you create are readable by your group, but not readable or writable by any other user. To have new files have a different permission, you can set your umask, which determines the permissions of newly created files and folders. To see what umask is in effect, type
umask -S. The default permission, which restricts access to members of your lab, is given by
umask u=rwx,g=rx,o= which removes “other” access and gives the group owner read and execute access. To change your umask to allow group write permissions, type
umask u=rwx,g=rwx,o=. To change your umask to allow “other” read access, type
In order for your new umask to be permanent, add one of the above umask commands to the login script in your home directory. This is typically .bash_profile or .bashrc (note the leading dot in the file names) but differs if you use a shell other than bash. For example, to change your umask to allow “other” read access, type the following to append the umask command to your .bashrc:
echo "umask u=rwx,g=rx,o=rx" >> ~/.bashrc
It is recommended not to use ACL’s, but to instead use supplemental permission groups whenever possible.
Kamiak has NFSv4 Access Control Lists (ACLs) enabled which can be managed with the commands
nfs4_getfacl. Users who are interested in utilizing nfs4_acl should be aware that this type of ACL is distinct from POSIX ACLs. More information about ACLs can be found at this site. Complete documentation can be found by running
man nfs4_acl. Let’s look at a very simple example of adding an ACL to an existing directory to give a user in another lab access to our data:
$ ls -ld example/ drwxrwx--- 2 my.NID its_p_sys_ur_kam-its 4096 Aug 10 08:14 example/ $ nfs4_getfacl example A::OWNER@:rwaDxtTnNcCy A:g:GROUP@:rwaDxtTnNcy A::EVERYONE@:tcy $ nfs4_setfacl -a A:df:email@example.com:RWX example/ $ nfs4_getfacl example A:fd:firstname.lastname@example.org:rwaDxtTnNcCy A::OWNER@:rwaDxtTnNcCy A:g:GROUP@:rwaDxtTnNcy A::EVERYONE@:tcy
If we provide standard permissions of RWX (must be capitalized)
nfs4_setfacl will convert it into the ACL format automatically. Initially the directory was only accessible by the user owner and members of the lab group. We added another user to be able to read and write data within the directory. We also included the inheritance flags ‘f’ and ‘d’ so this ACL entry will be added to any new files and directories that are created. To the the same with a group instead of a single user, we also include the ‘g’ flag:
$ nfs4_setfacl -a A:dfg:email@example.com:RWX example/ $ nfs4_getfacl example A:fdg:firstname.lastname@example.org:rwaDxtTnNcCy A:fd:email@example.com:rwaDxtTnNcCy A::OWNER@:rwaDxtTnNcCy A:g:GROUP@:rwaDxtTnNcy A::EVERYONE@:tcy
man nfs4_acl to see further documentation on ACLs. Note that you will need to specify users and groups as whateverName@ad.wsu.edu.