File Transfers

scp and rsync

You can transfer files between your workstation and the cluster on the command line using the scp command. This command behaves much like the basic Linux cp command, except you may use a remote address as the source or destination file. The syntax is as follows:

scp source_file destination_file

The following command will copy the file testfile from the /home/wsuNID/ directory on the remote server kamiak.wsu.edu to your workstation’s local directory “.” (a period represents the current working directory).

user@localhost> scp remoteuser@kamiak.wsu.edu:/home/remoteuser/testfile .

To copy a directory along with all its contents you will need to add the -r recursive flag. The following command will copy the simdata directory and all its contents to your local machine.

user@localhost> scp -r remoteuser@kamiak.wsu.edu:/home/remoteuser/simdata .

A powerful alternative to scp is rsync. Full documentation can be found by running man rsync. An example of a recursive copy (like the `scp -r …` above) is:

user@localhost> rsync -ravxS remoteuser@kamiak.wsu.edu:/home/remoteuser/simdata .

The “S” option is equivalent to ‐‐sparse, which will ensure that sparse files are transferred correctly.

It is important to note that a trailing slash, or lack thereof, changes rsync’s behavior. If the source directory is written with a trailing slash (remoteuser@kamiak.wsu.edu:/home/remoteuser/simdata/) then rsync will transfer the contents of that directory to the destination. Without a trailing slash rsync will transfer the directory itself and its contents.

Graphical file transfer software

There are many graphical file transfer solutions available. The following are the three most popular and are fairly intuitive. Regardless of your choice of software, you must connect to the cluster using the Secure File Transfer Protocol (SFTP).

• Filezillla is available on all platforms. Be careful when downloading and installing as the hosting site, sourceforge, has begun to bundle bloatware with its downloads.
• Fetch is a full-featured file transfer client for Mac and is free to the academic community.
• WinSCP is a free Windows client.

Globus

In addition to scp and rsync you can move files by Globus. Kamiak has a Globus Endpoint named wsuhpc#kamiak, which provides read-only access to /data folders and read-write access to /scratch folders you created.

We currently don’t integrate with Globus’ SSO authentication so you’ll need to sign up for a Globus ID or link another account (e.g. Google) and log into globus.org with that. Then search for the endpoint and authenticate with your WSU username and password when it asks for it.

Once that is done and you are logged in it should bring you to a page that asks you to select an “endpoint”. There is an endpoint at the source and an endpoint at the destination of each file transfer. Kamiak hosts a Globus Connect server, so that’s one endpoint. Your laptop or workstation will be the other and you will need to install Globus Connect Personal onto it. After you sign into Globus Online there is a link in the lower right corner which says “Get Globus Connect Personal”, click that or click here. Choose a name for your endpoint, copy the key the site provides, then download and install the software. After running it and giving it the key when prompted, you should then have a Globus Connect Personal endpoint running.

Now we need to go back to the Globus Web page where it asks you to select endpoints. Click to select an endpoint then search for “kamiak”. You should see our endpoint come up: wsuhpc#kamiak. Once you select that, it should prompt you to log in again. This time you’ll need to log in with your WSU username and password. The reason for that is you now are authenticating to Kamiak. Do that and you should see the directories “data”, and “scratch”. That is Kamiak’s storage.

Next, on the other side where it asks you to select an endpoint, search for the one you just created using the name you entered for it. It may also be under the “Administered by Me” tab just under the search bar.

You should now be able to move data between Kamiak and your endpoint.

Please note that /data folders accessible under globus are read-only, while /scratch is read-write for any scratch folders you created.