Running e4s-cl on Crusher at OLCF#

Available on Crusher is a Slurm-based job facility, the Singularity/Apptainer container system and a CRAY-based MPI environment. This page will outline how to setup, prepare and run a job using e4s-cl on this particular system.

Setting up with a profile#

Initialize e4s-cl using the init command.

Make sure you load a MPI module compatible with the MPICH ABI. On crusher, they often are named cray-mpich-abi/x.y.z.

The program will run a simple MPI program and observe its execution to determine what libraries it used. Make sure you update the --launcher_args option value to something you would use on the system, as these options will be passed to srun when launching the process. Available fields are listed on the crusher help page.

$ e4s-cl init                            \
     --backend singularity               \
     --image $YOURIMAGE                  \
     --launcher srun                     \
     --launcher_args '-n 2 -N 2 ... '    \

Creating an initialization script#

Next, we need to make sure the required libraries will be loaded in the container. In this example, we will load the trilinos library module using spack.

In a new file, write what should be done to load the required modules:

# Setup spack
. /spack/share/spack/

# Load trilinos
spack load --first trilinos

Then link this file to the selected profile:

$ e4s-cl profile edit --source $PWD/

Running jobs#

You can run a e4s-cl job like a regular Slurm job by adding a e4s-cl launcher command before the srun directive. This can be done using a job file:

$ cat run.job
#SBATCH -N 2 -t 00:30:00

e4s-cl srun -n 2 /path/to/executable
$ sbatch run.job

Or from the command line in an interactive job:

$ e4s-cl srun -n 2 -N 2 -t 00:30:00 /path/to/executable