A1-to-A2 Data Migration

A computer in the HPC cluster is referred to as a node, and there are two types of nodes on Andromeda: login nodes and compute nodes.

Pre-Copied A1 Data

Before copying data from A1 yourself it’s important to check a002:/migration/data to see if your data has been pre-copied for you.

• A lot of user data has already been copied to A2 under the a002:/migration/data directory, which is a temporary read-only location during the migration. Please check there first to see if your data has already been copied from A1 for you.
  • The data there is read-only, so please don’t try to modify it or use it as part of a slurm job.
  • This data is currently being refreshed from A1 daily.
  • You can simply copy from /migration/data/<login> to your home directory or project directory using standard tools.
    • For example: using the “cp -pr” command
    • -p preserves permissions, ownership, and timestamps
    • –r means copy data recursively, including subdirectories
• A1:/data/projects/<project> had an initial sync to A2:/projects earlier this year
  • If you’d like to have your project’s data re-sync’d to A2, please submit a request to https://bc.edu/researchhelp

• In order to better manage data on A2 there are quotas limiting how much data can be stored in each space available to you.
  • Each user’s home directory has a quota of 50GB
    • This quota cannot be increased.
  • Each scratch and project directory has a quota limit of 1TB each
    • If you need more capacity in either /scratch or /projects, please submit a request to https://bc.edu/researchhelp

Manually Copy Data from A1 -> A2

If you aren’t able to find your A1 data under /migration/data, then the rsync command can be used to easily copy data from A1 → A2. Here are instructions for doing so.

In the following example the user, represented by the environment variable $USER, will copy the directory called my_data, which is located under their home directory, to their home directory on a002 (Andromeda 2). The last character in the following command is a tilde “~”, which represents the home directory on the a002 remote host.

[l001 ~]$ rsync -avuz $HOME/my_data $USER@a002:~

Here is some more info about the command.

• -a: Archive mode; preserves permissions, timestamps, and symbolic links.
• -v: Verbose mode; shows you what files are being transferred.
• -u: Ensures that newer files on the destination do not get overwritten.
• -z: Compresses data during transfer, which can speed things up.
• $HOME/my_data: The directory or file you want to copy.
  • Make sure to include a trailing slash if you want to copy the contents of the directory, not the directory itself.
• $USER@a002: Your username and the hostname of the Andromeda 2 login node.
• $USER@a002:/home/$USER: The directory where you want to copy the data on the destination node.

*If you are copying a large amount of data, consider running the rsync command within a screen or tmux session to prevent interruptions.

Setting Up Passwordless SSH (optional)

Passwordless SSH lets you log in without typing your password every time, making rsync transfers more convenient.

1. Generate SSH Keys (Only if you don’t have them already. To check if you have them, you can list the contents of $HOME/.ssh).

On a001, run:

ssh-keygen -t ed25519

• Press Enter to accept the default file location (~/.ssh/id_ed25519).
• When prompted for a passphrase, you can either enter one (for extra security) or press Enter twice to leave it blank. 

ssh-copy-id $USER@a002

2. Test the Connection:

• If you set up everything correctly, you should be logged in without entering your password.

ssh $USER@a002 hostname

It should return the following:

a002.m31.bc.edu