Using the queue system on skd-cyclone

General information

Purpose

The machine skd-cyclone.klientdrift.uib.no is used as a general-purpose linux computer with several connected storage. Researchers, PostDocs, PhD students and Master students use the machine for calculations. On several occasions, this has lead shortages in calculation resources, concerning both memory and CPU. The machine has started to "swap" memory to the disk, slowing down all calculations on the computer to an extend where it can not be used any more.

The queuing system on skd-cyclone is in place to schedule jobs according to the available resources on the machine. This leads to a more balanced load of the machine, and avoids swapping situations.

Advantages

The main advantage of the queue system is that is makes calculations on skd-cyclone more stable and reliable, and thus overall faster.

Further advantages are:

Interactive run of programs still possible
Lightweight wrapper scripts required to run jobs on queue
Email on fail/completion of job
Queues with different priority
The queue at GFI is a way of introducing already MSc students to requirements of a HPC environment

Disadvantages

No use of screen or similar commands to interactively connect to running jobs
Waiting time when queue is full
Need to write "wrapper script" (see below)

Rules and procedures

The queue system is opt-in basis. That means that there is no obligation to use the queue system. Small jobs, for example, can run directly on the machine. Users who repeatedly run large jobs, however, such as NWP models or diagnostic code, will be asked to submit their job to the queue system.

Short tutorial

1. create a basic wrapper script for your job

  #!/bin/bash
  #
  # SLURM wrapper script
  # start this job with sbatch NAME_OF_THIS_FILE directly on the machine
  #
  #SBATCH --job-name=my_slurm_job
  #SBATCH --workdir=/scratch/$USER
  #SBATCH --partition=normal
  #SBATCH --output=results.%j.txt
  #SBATCH --error=errors.%j.out
  #SBATCH --mail-type=ERR,END
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=4

  #apply any required path settings
  export PATH=$PATH:/Data/gfi/met/bin

  #change to your working directory
  cd /Data/gfi/met/batch

  # command to start your job
  ./run_flexpart_noresm.sh

  # done

2. submit the script to the queue

sbatch ./my_queue_job.sh

3. monitor your jobs and the general load on the queue

squeue

4. examine what resources are used by your job

sstat <jobid>

5. cancel a job if required

scancel <jobid>

6. receive an email when the job is finished

For example with the following header:

From: slurm@klientdrift.uib.no
To: <username>@uib.no
Subject: SLURM Job_id=34 Name=WaterSip3.0 Ended, Run time 00:01:00

Converting a script to a queue script

add elements to the header

Description of each element

The SLURM queue is controlled by a set of parameters at the beginning of a shell script:

#SBATCH --<parameter>=<argument>

These parameters control for example the output directory, emails, resource allocation and emailing.

The most important aspect is to determine the memory demands of your job. This is nicely described here: Finding out which memory limits to use

The most common script parameters are explained here in short:

mem: how much memory your job will use at most (in MB).

Example: --mem 1000

workdir: output files will be created in this directory.

Example: --workdir=/scratch/$USER

queue: which queue the output should be processed on.

Example: --partition=default

job-name: provide a meaningful name to your job that will be visible on the queue

Example: --job-name=my_slurm_job

output: normal output will be written to this file. %j is a placeholder for the jobid

Example: --output=results.%j.txt

error: error messages will be written to this file. %j is a placeholder for the jobid

Example: --error=errors.%j.out

mail-type: an email will be send on completion (parameter END) or error (parameter ERR)

Example: --mail-type=END

time: maximum requested computation time (12h in the example below)

Example: --time=12:00:00

ntasks: number of cores requested, for example for a MPI parallelisation

Example:--ntasks=1

cpus-per-task: number of parallel threads to be run, for example for OpenMP parallelisation

Example: --cpus-per-task=4

mem-per-cpu: requested memory per CPU (100MB in the example below)

Example: --mem-per-cpu=100

Best practices, Q&A

Further information

Contacts

In case of questions, please contact Idar Hessevik or Harald Sodemann at GFI

Links

A very readable blog about the SLURM queue

A easy-to-understand SLURM documentation

The full SLURM documentation