Run jobs on SCARF

This feature is currently in beta. If you encounter any issues while trying to run jobs on SCARF, check the console output for errors. If you cannot resolve the issue from there, please contact us.

As part of a freestyle job, you can configure the job to run on SCARF. The script and options that you input will be packaged into a script, which is submitted to SCARF when the Jenkins job is run. The Jenkins job will remain running until the SCARF job is complete, and then recover output files.

Running on SCARF node

First, you must make sure you run the job on a SCARF agent. Enter the label 'scarf' under Restrict where this job can be run at the top of the freestyle job configuration.

Restricting job agent
Restricting job to run on the SCARF agent.

Build step

Note: We recommend that you do not use other build steps in conjunction with this step. If you do use any other build steps as part of your job, make sure these steps appear before the build step for submitting the SCARF job. No further steps can be run after the SCARF job build step is complete.

Under Build, select Run SLURM script:

Selecting SLURM option
Selecting to run job on a SLURM system such as SCARF.

The following interface will appear:

SLURM job settings
Build options for SLURM job.

Script

This is the script that will be run on SCARF. The list of available modules is different to other nodes on Anvil - see the SCARF software page for information. While the script output will be accessible in Anvil, it is not the quickest way to troubleshoot - if you have an existing SCARF account, we recommend you log into SCARF yourself and check that the script you enter will run properly.

Note that limits may have been placed on resource usage by the administrators, including the number of nodes that may be requested, and the total CPU time that can be requested for a single job. You will be warned when you try to run the job if your requested resources exceed these limits.

Finally, do not enter SLURM options in this script. Any lines beginning with #SBATCH will be filtered out before submission to SCARF. Please use the options below to specify your resource requirements.

Nodes

The number of nodes you require. Corresponds to the -N option in SLURM.

Number of tasks

The number of tasks to run. Corresponds to the -n option in SLURM.

CPUs per task

The number of CPUs required for each task. Corresponds to the -c option in SLURM. The scarf18 and scarf17 hardware on SCARF has 24 CPUs per node (older hardware has fewer - see the hardware page of the SCARF documentation for more details).

Walltime required

The amount of walltime required for the job. Note that this must be entered in minutes; formats such as hh:mm:ss are not accepted. Corresponds to the -t option in SLURM.

Advanced options

SLURM job advanced settings
Advanced build options for SLURM job.

These settings are optional; if left empty, they will revert to default values.

Queue The queue, or partition, to use for the job. Corresponds to the -p option in SLURM. Default is the scarf partition.

Node features

Any features you require the nodes to have. Include all punctuation characters, but do not surround the input with quotes, e.g. [scarf17|scarf18], not "[scarf17|scarf18]". Corresponds to the -C option in SLURM. Default is to request no specific features.

Exclusive mode

Require that the nodes reserved for this job are not running any other job at the same time. Enabling this option is recommended. Corresponds to the --exclusive option in SLURM. Default is for this mode to be off.

Files to recover

If the job writes information to files, and you want to retrieve those files, enter the file names here, separated by commas. The default SLURM output of the form slurm-jobID* will always be recovered.

Running the job

Run the Jenkins job in the normal way. The console output will provide some information about the progress of the job. Outputs from the sbatch command and the SCARF job itself are recovered as 'build artifacts', which can be viewed on the build page. Some or all of these artifacts should appear even if the build fails; if the reason for failure is not clear from the console output, check these artifacts for more information.