Slurm

Aus PC2 Doc
Wechseln zu: Navigation, Suche

Workload Manager - SLURM

The Noctua1 system uses the Simple Linux Utility for Resource Management (SLURM)

  • plenty of documentation can be found on https://slurm.schedmd.com/documentation.html
  • daily work commands are
    • sbatch - submit a batch script to SLURM
    • srun - run parallel job
    • scancel - signal Job under the control of SLURM
    • squeue - information about running
    • sinfo - info about the partions
  • the entire information about your program execution is contained in a batch script which is submitted via sbatch
  • the batch script contains one or more parallel jobs runs executed via srun (job step).
  • SLURM of Noctua1 assigns nodes exclusive to a user (no space sharing).


SLURM System Information

  • to get information about the system use sinfo
> sinfo
compute*     up   infinite      2   comp cn-[0235,0243]
compute*     up   infinite    232  alloc cn-[0001-0008,0011-0015,0020,0029-0037,0039-0051,0053-0136,0138-0179,0181-0211,0213-0229,0232-0234,0237-0242,0244-0256]
compute*     up   infinite     22   idle cn-[0009-0010,0016-0019,0021-0028,0038,0052,0137,0180,0212,0230-0231,0236]
fpga         up    2:00:00     16   idle fpga-[0001-0016]
all          up   infinite      2   comp cn-[0235,0243]
all          up   infinite    232  alloc cn-[0001-0008,0011-0015,0020,0029-0037,0039-0051,0053-0136,0138-0179,0181-0211,0213-0229,0232-0234,0237-0242,0244-0256]
all          up   infinite     38   idle cn-[0009-0010,0016-0019,0021-0028,0038,0052,0137,0180,0212,0230-0231,0236],fpga-[0001-0016]


Launching applications

srun is the application launcher

  • It should be used to run application on Noctua1 nodes - interactively or in a Batch Job
  • If srun is not used, the application is launched on the frontend node (don't use the frontends for compute intensive applications!!)
  • srun launches groups of tasks on the nodes
  • some important parameters to set are:
Option Description
-n, --ntasks total number of tasks
--ntasks-per-node number of tasks per compute/fpga node
-N, --nodes number of nodes
-t, --time Walltime - maximum execution time
  • the parameters can also be specified via #SBATCH in batch script
  • threading has to be controled via OMP_NUM_THREADS
#!<your_shell>
...
#SBATCH -N 4

export OMP_NUM_THREADS=1
srun -n 160 --tasks-per-node=40 ./<exe>

export OMP_NUM_THREADS=20
srun -n 8 --tasks-per-node=2 ./<exe>
  • more details on controling the task mapping on the hardware can be found in Noctua-Tuning


Monitoring SLURM Jobs

  • While running you can inspect your job with squeue
> squeue -u $USER
  JOBID   PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
  12345   compute       ...  
  • for more information you can use scontrol show job <JOBID> from an interactive session to get the job steps
  • use sacct to get accounting information about active and completed jobs
  • In case of a problem with your job (e.g. no or unexpected output), you can cancel it
scancel <JOBID>


Remark:

For massive file I/O the LUSTRE file system must be used ($PC2PFS/<group>)