Transfer Data To/From Andromeda 

One of the most common questions is how to transfer files to or from an HPC cluster. Users can use one of the following graphics transfer tools and command line tools.

Graphical Transfer Tools: cyberduck

Cyberduck Setup Instruction – Andromeda 

  1.  Download Cyberduck from: https://cyberduck.io/download/. If you are not on campus, you will need to install and login to EagleVPN before  opening a connection: https://bcservices.bc.edu/service/vpn 
  2.  Start Cyberduck, click Open Connection on the top left corner 
  1. Select SFTP (SSH File Transfer Protocol), fill in the Server andromeda.bc.edu, and leave the Port as 22. Enter your BC username and password, then click Connect.
  2. If  you are asked to allow an unknown fingerprint, click Allow and check  the box Always.
  1. Now you are connected and you can see all your files in the cluster. 

Command Line Tools. Rsync and scp

Rsync  – A Fast, Versatile, Remote (and Local) File-copying Tool

Rsync is well-known for its transfer algorithm, which reduces the amount of data sent over the network by only sending the missing files between the source and the existing files in the destination. It is frequently applied for remote transfers between two systems and is also great for local work.

The basic syntax of the Rsync command is:

rsync  [source] [destination]

Here, “source” and “destination” are the paths of the files or directories being copied.

If you are running a large transfer with Rsync, it takes a while, please put it inside a tmux virtual login session. This avoids your transfer from network interruptions between your computer and the transfer node.

https://www.redhat.com/sysadmin/introduction-tmux-linux

Using Rsync for local work

Copy “myfile” to “myfile-copy” with progress meter:

rsync –progress myfile myfile-copy

Copy “mydir” to “mydir-copy”

rsync -avP mydir/ mydir-copy/

Here, -a indicates archive mode, which preserves ownership, permissions, and creation/modification times. The -v option enables verbose mode, printing the name of every file, and -P displays a progress bar.

Using Rsync to Sync with a remote system

Upload my directory or my file from the local machine to the user johnchris home directory on the andromeda cluster

For example:

rsync -avxz mydir/ johnchris@andromeda.bc.edu:~/mydir/

Download my directory or file from user johnchris home directory on the andromeda cluster to the local machine

For example:

rsync -avz  johnchris@andromeda.bc.edu:~/mydir/ .

SCP – Secure Copy (Remote File Copy Program)

We recommend using SCP to copy data to and from the BC cluster Andromeda. It’s a fast and simple application for securely copying data in a one-time transfer.

The general syntax of the SCP command is:

scp [options] [source] [destination]

Here, “options” can include things like the username and hostname of the source or destination, and “source” and “destination” specify the paths of the files or directories being copied.

Example: Transfer File or a Folder from Your Local Computer to the BC Cluster

scp [location of file] [username@server:][destination of file]

A cluster user with the login id johnchris would like to transfer a ‘ mydata.txt ‘ file from their personal laptop to the ‘/scratch/johnchris’ directory on the andromeda cluster.

scp mydata.txt johnchris@andromeda.bc.edu:/scratch/johnchris

Add -r option, if Transferring a whole folder named ‘mydir’

scp -r mydir johnchris@andromeda.bc.edu:/scratch/johnchris

Example: Transfer a File or a folder from the BC Cluster to Your Local Computer

scp johnchris@andromeda.bc.edu:/data/johnchris/ mydata.txt .

To transfer the ‘mydirectory’ directory and all of its contents with the ‘r’ (recursive) option

scp -r johnchris@andromeda.bc.edu:/data/johnchris/mydata.txt .

“.” represents your current working directory.

To specify the destination, simply replace the “.” with the full path:

scp -r johnchris@andromeda.bc.edu:/data/johnchris/ mydata.txt . /path/mydir

Scroll to Top