First, login to Andromeda:
ssh loginname@andromeda.bc.edu (where “loginname” is your user name)
Load an interactive session by typing into the command line “interactive”.
Gaussian Modules:
To see what modules are available to be loaded, we can simply type “module avail”. A list of modules such as the one below should show up:
There are four versions of Gaussian available (gaussian/16.C01_AVX2.Linda, gaussian/16.C01_AVX2, gaussian/16.C01_SSE4, and gaussian/16.C01_SSE4.Linda). The module “gaussian/16.C01_AVX2” is set by default (D). AVX2 (Advanced Vector Extensions 2) and SSE4 (Streaming SIMD Extensions 4) are simply two different instruction sets for Intel processors.
Non-Linda versions of Gaussian support only shared memory parallelization, i.e., parallel execution within a single node
Linda: supports distributed memory parallelization. This is essential for running jobs across multiple nodes in a cluster.
Example of Geometry Optimization using Gaussian 16
This can be done on SLURM, either using single-node/single-threads, or alternatively using single-node/multi-threads, or multi-nodes/multi-threads. This document will discuss all of these cases, section by section.
Checkpoint file:
To ensure job reliability and recovery, all users are required to specify a checkpoint file (.chk) when submitting Gaussian jobs on Andromeda. The checkpoint file allows you to restart or recover your job in case of failure, minimizing the need to rerun calculations from scratch.
Single-node/Multi-Thread with Gaussian Default Module
Create the following input file (named 1_3_S_Dioxane.com):
%NProcShared=32
%mem=64gb
%chk=1_3_S_Dioxane.chk
#p opt=verytight M062X/aug-cc-pVTZ SCF=(MaxCycle=100) Int(grid=175974)
1,3-Dioxane ion gas phase
0 1
C -1.74085610 -2.79544280 0.00000000
C -1.73858810 -0.57982780 1.16066100
C -3.25371310 -0.57916680 1.16017200
C -3.80651310 -1.98978780 1.15887600
H -1.36830710 -3.34107680 0.90656200
H -1.36556210 -3.34530180 -0.90191000
H -1.36605310 -1.01467280 2.12527200
H -1.36256510 0.47430920 1.09866600
H -3.62903310 -0.02973880 2.06228600
H -3.62556810 -0.03256580 0.25384900
H -3.54050910 -2.49550980 2.12419100
H -4.92506110 -1.95472680 1.09393800
O -3.25596210 -2.79544280 0.00000000
O -1.18892510 -1.38436480 0.00000000
(When creating the above script, you will need to make sure, in your vim file, that there is a BLANK LINE after the last line- otherwise, when you submit the job, you will encounter an error message like the one below:)
In this file, we have specified the total number of shared processors (32) and total memory requested (64) in order to run this job across multiple threads.
Create the following file called 1_3_S_Dioxane.sl:
#!/bin/bash
#SBATCH --job-name=1_3_Dioxane
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=8 ## number> 1, for multithreading
#SBATCH --mem=64GB
#SBATCH --time=48:00:00
#SBATCH --output=1_3_S_Dioxane.o
#SBATCH --error=1_3_S_Dioxane.e
#SBATCH --partition=medium
# Load default gaussian module of gaussian/16.C01_AVX2
module add gaussian
# Change to working directory
cd $WORKDIR ### or cd to whatever directory you have your input and .sl files are in
# Set Gaussian scratch directory
export GAUSS_SCRDIR=/scratch/$USER
# Check if GAUSS_SCRDIR exists and is a file
if [ -f "$GAUSS_SCRDIR" ]; then
echo "The Gaussian scratch directory, $GAUSS_SCRDIR, exists and is a file. Please rename and try again."
exit 1
# Check if GAUSS_SCRDIR does not exist, then create it
elif [ ! -d "$GAUSS_SCRDIR" ]; then
mkdir -p "$GAUSS_SCRDIR"
fi
# Run Gaussian calculation
g16 1_3_S_Dioxane.com
Submit this job using the following command: sbatch 1_3_S_Dioxane.sl
Multi-Node/Multi-Thread with Gaussian Linda Version
One can use the following Gaussian input file (1_3_M_Dioxane.com)
%NProcShared=128
%mem=128gb
%chk=1_3_M_Dioxane.chk
#p opt=verytight M062X/aug-cc-pVTZ SCF=(MaxCycle=100) Int(grid=175974)
1,3-Dioxane ion gas phase
0 1
C -1.74085610 -2.79544280 0.00000000
C -1.73858810 -0.57982780 1.16066100
C -3.25371310 -0.57916680 1.16017200
C -3.80651310 -1.98978780 1.15887600
H -1.36830710 -3.34107680 0.90656200
H -1.36556210 -3.34530180 -0.90191000
H -1.36605310 -1.01467280 2.12527200
H -1.36256510 0.47430920 1.09866600
H -3.62903310 -0.02973880 2.06228600
H -3.62556810 -0.03256580 0.25384900
H -3.54050910 -2.49550980 2.12419100
H -4.92506110 -1.95472680 1.09393800
O -3.25596210 -2.79544280 0.00000000
O -1.18892510 -1.38436480 0.00000000
(Again, make sure there is an extra blank line after the last line in this file)
Similarly, the following submission script (call it 1_3_M_Dioxane.sl) will run the above input file:
#!/bin/bash
#SBATCH --job-name=1_3_Dioxane
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=8 #set to >1 for multithreading
#SBATCH --mem=128GB
#SBATCH --time=48:00:00
#SBATCH --output=1_3_M_Dioxane.o
#SBATCH --error=1_3_M_Dioxane.e
#SBATCH --partition=medium
## Load the gaussian/16.C01_SSE4.Linda -- parallel version
module load gaussian/16.C01_AVX2.Linda
# Change to working directory
cd $WORKDIR ### if you have already set $WORKDIR in your bashrc; otherwise cd to whatever directory your input and .sl files are in
# Set Gaussian scratch directory
export GAUSS_SCRDIR=/scratch/$USER
# Check if GAUSS_SCRDIR exists and is a file
if [ -f "$GAUSS_SCRDIR" ]; then
echo "The Gaussian scratch directory, $GAUSS_SCRDIR, exists and is a file. Please rename and try again."
exit 1
# Check if GAUSS_SCRDIR does not exist, then create it
elif [ ! -d "$GAUSS_SCRDIR" ]; then
mkdir -p "$GAUSS_SCRDIR"
fi
# Run Gaussian calculation
g16 1_3_M_Dioxane.com
Submit this job using the following command: sbatch 1_3_M_Dioxane.sl
Resubmit Job with Checkpoint file
1.Checking if the calculation has finished appropriately
For example, open the file “1_3_S_Dioxane.out”- the output file- then type the command “/Normal termination of Gaussian 16 at” and it will lead you to the part of the output where the phrase “Normal termination of Gaussian 16 at” occurs. After this keyword, there should be the day of week, month, day of month, time, and year the calculation finished (e.g. Tue Nov 14 15:31:56 2017))
2. Restarting a Geometry Optimization
As long as you set a *.chk file inside the Gaussian input file, it will generate a checkpoint file after the calculation.
For example: when running 1_3_S_Dioxane.com, you generate a .chk file named “1_3_S_Dioxane.chk”. If your calculation terminates prematurely, we can use the .chk file to restart your calculation.
In order to restart it, go to your original file 1_3_S_Dioxane.com to change the following line:
#p opt=verytight M062X/aug-cc-pVTZ SCF=(MaxCycle=100) Int(grid=175974)
To:
#p opt=(verytight, Restart) M062X/aug-cc-pVTZ SCF=(MaxCycle=100) Int(grid=175974)
Then, you can use the same .sl file you used previously to re-submit the job. Note that while this will restart the calculation, this will overwrite the results of the previous calculation (1_3_S_Dioxane.out) with the results of the restarted calculation (unless if, in your .sl file, you change the value of the parameter. #SBATCH –output=1_3_Dioxane.o to something different from your first run.