Introduction to SLURM & SLURM Batch Scripts Overview
undefined
I
n
t
r
o
d
u
c
t
i
o
n
t
o
S
L
U
R
M
&
S
L
U
R
M
b
a
t
c
h
s
c
r
i
p
t
s
Anita Orendt
Assistant Director
Research Consulting & Faculty Engagement
O
v
e
r
v
i
e
w
o
f
T
a
l
k
•
Basic SLURM commands
•
Accounts and Partitions
•
SLURM
batch directives
•
SLURM Environment Variables
•
SLURM Batch scripts
•
Running an Interactive Batch job
•
Monitoring Jobs
•
Where to get more Information
B
a
s
i
c
S
L
U
R
M
c
o
m
m
a
n
d
s
•
s
i
n
f
o
-
s
h
o
w
s
p
a
r
t
i
t
i
o
n
/
n
o
d
e
s
t
a
t
e
•
s
b
a
t
c
h
<
s
c
r
i
p
t
n
a
m
e
>
-
l
a
u
n
c
h
e
s
a
b
a
t
c
h
s
c
r
i
p
t
•
s
q
u
e
u
e
-
s
h
o
w
s
a
l
l
j
o
b
s
i
n
t
h
e
q
u
e
u
e
–
s
q
u
e
u
e
-
u
<
u
s
e
r
n
a
m
e
>
-
s
h
o
w
s
o
n
l
y
y
o
u
r
j
o
b
s
•
s
c
a
n
c
e
l
<
j
o
b
i
d
>
-
c
a
n
c
e
l
s
a
j
o
b
Notes:
F
o
r
s
i
n
f
o
,
s
q
u
e
u
e
–
c
a
n
a
d
d
–
M
a
l
l
t
o
s
e
e
a
l
l
c
l
u
s
t
e
r
s
u
s
i
n
g
g
i
v
e
n
s
l
u
r
m
i
n
s
t
a
l
l
a
t
i
o
n
(
n
o
t
c
h
p
e
a
k
,
k
i
n
g
s
p
e
a
k
,
l
o
n
e
p
e
a
k
,
a
s
h
)
C
a
n
a
l
s
o
a
d
d
–
M
c
l
u
s
t
e
r
O
R
u
s
e
f
u
l
l
p
a
t
h
/
u
u
f
s
/
<
c
l
u
s
t
e
r
>
.
p
e
a
k
s
/
s
y
s
/
p
k
g
/
s
l
u
r
m
/
s
t
d
/
b
i
n
/
<
c
o
m
m
a
n
d
>
t
o
l
o
o
k
a
t
t
h
e
q
u
e
u
e
,
o
r
s
u
b
m
i
t
o
r
c
a
n
c
e
l
j
o
b
s
f
o
r
a
d
i
f
f
e
r
e
n
t
c
l
u
s
t
e
r
Redwood has own slurm setup, separate from others
S
o
m
e
U
s
e
f
u
l
A
l
i
a
s
e
s
•
Bash to add to .aliases file:
alias si="sinfo -o \"%20P %5D %14F %8z %10m %10d %11l %16f %N\""
alias si2="sinfo -o \"%20P %5D %6t %8z %10m %10d %11l %16f %N\""
alias sq="squeue -o \"%8i %12j %4t %10u %20q %20a %10g %20P %10Q
%5D %11l %11L %R\""
•
Tcsh to add to .aliases file:
alias si 'sinfo -o "%20P %5D %14F %8z %10m %11l %16f %N"'
alias si2 'sinfo -o "%20P %5D %6t %8z %10m %10d %11l %N"'
alias sq 'squeue -o "%8i %12j %4t %10u %20q %20a %10g %20P %10Q %5D
%11l %11L %R"‘
C
a
n
a
d
d
–
M
t
o
s
i
a
n
d
s
q
a
l
s
o
You can find these on the CHPC Slurm page
https://www.chpc.utah.edu/documentation/software/slurm.php#aliases
A
c
c
o
u
n
t
s
&
P
a
r
t
i
t
i
o
n
s
•
Y
o
u
n
e
e
d
t
o
s
p
e
c
i
f
y
a
n
a
c
c
o
u
n
t
a
n
d
a
p
a
r
t
i
t
i
o
n
t
o
r
u
n
j
o
b
s
•
You can see a list of partitions using the sinfo command
•
For general allocation usage the partition is the cluster name
•
If no allocation (or out of allocation) use
clustername
-freecycle for partition
•
Your account is typically your PI’s name (e.g., if your PI is Baggins, use the
"baggins" account) – there are a few exceptions!
•
Owner node accounts and partition have the same name – PI last name with
cluster abbreviation, e.g., baggins-kp, baggins-np, etc
•
Owner nodes can be used as a guest using the "owner-guest" account and
the
cluster
-guest partition
•
Remember general nodes on notchpeak need allocation; general nodes on
kingspeak and lonepeak are open to all users without allocation
•
PE has its own allocation process
M
o
r
e
o
n
A
c
c
o
u
n
t
s
&
P
a
r
t
i
t
i
o
n
s
See
https://www.chpc.utah.edu/documentation/guides/index.php#parts
Q
u
e
r
y
y
o
u
r
a
l
l
o
c
a
t
i
o
n
~]$ myallocation
You have a
general
allocation on
kingspeak
. Account:
chpc
, Partition:
kingspeak
You have a
general
allocation on
kingspeak
. Account:
chpc
, Partition:
kingspeak-shared
You can use
preemptable
mode on
kingspeak
. Account:
owner-guest
, Partition:
kingspeak-guest
You can use
preemptable
GPU
mode on
kingspeak
. Account:
owner-gpu-guest
, Partition:
kingspeak-
gpu-guest
You have a GPU allocation on
kingspeak
. Account:
kingspeak-gpu
, Partition:
kingspeak-gpu
You have a
general
allocation on
notchpeak
. Account:
chpc
, Partition:
notchpeak
You have a
general
allocation on
notchpeak
. Account:
chpc
, Partition:
notchpeak-shared
You can use
preemptable
GPU
mode on
notchpeak
. Account:
owner-gpu-guest
, Partition:
notchpeak-
gpu-guest
You can use
preemptable
mode on
notchpeak
. Account:
owner-guest
, Partition:
notchpeak-guest
You have a
GPU
allocation on
notchpeak
. Account:
notchpeak-gpu
, Partition:
notchpeak-gpu
You have a
general
allocation on
lonepeak
. Account:
chpc
, Partition:
lonepeak
You have a
general
allocation on
lonepeak
. Account:
chpc
, Partition:
lonepeak-shared
You can use
preemptable
mode on
lonepeak
. Account:
owner-guest
, Partition:
lonepeak-guest
You can use
preemptable
mode on
ash
. Account:
smithp-guest
, Partition:
ash-guest
N
o
d
e
S
h
a
r
i
n
g
•
Use the shared partition for a given set of nodes (using normal account for that partition)
notchpeak* 2 2/0/0/2 2:16:2 768000 1700000 3-00:00:00 chpc,skl,c32,m768 notch[044-045]
notchpeak* 32 32/0/0/32 4:16:2 255000 700000 3-00:00:00 chpc,rom,c64,m256 notch[172-203]
notchpeak* 4 4/0/0/4 2:16:2 95000 1800000 3-00:00:00 chpc,skl,c32,m96 notch[005-008]
notchpeak* 19 19/0/0/19 2:16:2 191000 1800000 3-00:00:00 chpc,skl,c32,m192 notch[009-018,035-043]
notchpeak* 1 1/0/0/1 2:18:2 768000 7400000 3-00:00:00 chpc,skl,c36,m768 notch068
notchpeak* 7 7/0/0/7 2:20:2 191000 1800000 3-00:00:00 chpc,csl,c40,m192 notch[096-097,106-107,153-155]
notchpeak-shared 2 2/0/0/2 2:16:2 768000 1700000 3-00:00:00 chpc,skl,c32,m768 notch[044-045]
notchpeak-shared 32 32/0/0/32 4:16:2 255000 3700000 3-00:00:00 chpc,rom,c64,m256 notch[172-203]
notchpeak-shared 4 4/0/0/4 2:16:2 95000 1800000 3-00:00:00 chpc,skl,c32,m96 notch[005-008]
notchpeak-shared 19 19/0/0/19 2:16:2 191000 1800000 3-00:00:00 chpc,skl,c32,m192 notch[009-018,035-043]
notchpeak-shared 1 1/0/0/1 2:18:2 768000 7400000 3-00:00:00 chpc,skl,c36,m768 notch068
notchpeak-shared 7 7/0/0/7 2:20:2 191000 1800000 3-00:00:00 chpc,csl,c40,m192 notch[096-097,106-107,153-155]
•
In script:
–
#SBATCH --partition=cluster-shared
–
#SBATCH --ntasks=2
–
#SBATCH --mem=32G
•
If there is no memory directive used the default is that 2G/core will be allocated to the job.
•
Allocation usage of a shared job is based on the percentage of the cores and the memory
used, whichever is higher
https://www.chpc.utah.edu/documentation/software/node-sharing.php
O
w
n
e
r
/
O
w
n
e
r
-
g
u
e
s
t
•
CHPC provides heat maps of
usage of owner nodes by the
owner over last two weeks
•
https://www.chpc.utah.edu/usage
/constraints/
•
Use information provided to
target specific owner partitions
with use of constraints (more
later) and node feature list
S
L
U
R
M
B
a
t
c
h
D
i
r
e
c
t
i
v
e
s
#SBATCH --time 1:00:00
wall
time of a job (or -t) in hour:minute:second
#SBATCH --partition=name
partition to use (or -p)
#SBATCH --account=name
account to use (or -A)
#SBATCH --nodes=2
number of nodes (or -N)
#SBATCH --ntasks 32
total number of tasks (or -n)
#SBATCH --mail-type=FAIL,BEGIN,END
events on which to send email
#SBATCH --mail-user=name@example.com
email address to use
#SBATCH -o slurm-%j.out-%N
name for stdout; %j is job#, %N node
#SBATCH -e slurm-%j.err-%N
name for stderr; %j is job#, %N node
#SBATCH --constraint “C20”
can use features given for nodes (or -C)
S
L
U
R
M
E
n
v
i
r
o
n
m
e
n
t
V
a
r
i
a
b
l
e
s
•
Depends on SLURM Batch Directives used
•
Can get them for a given set of directives by using the
env command inside a script (or in a srun session).
•
Some useful environment variables:
–
$SLURM_JOB_ID
–
$SLURM_SUBMIT_DIR
–
$SLURM_NNODES
–
$SLURM_NTASKS
B
a
s
i
c
S
L
U
R
M
s
c
r
i
p
t
f
l
o
w
1.
Set up the #SBATCH directives for the scheduler to request
resources for job
2.
Set up the working environment by loading appropriate modules
3.
If necessary, add any additional libraries or programs to $PATH
and $LD_LIBRARY_PATH, or set other environment needs
4.
Set up temporary/scratch directories if needed
5.
Switch to the working directory (often group/scratch)
6.
Run the program
7.
Copy over any results files needed
8.
Clean up any temporary files or directories
B
a
s
i
c
S
L
U
R
M
s
c
r
i
p
t
-
b
a
s
h
#!/bin/bash
#SBATCH --time=02:00:00
#SBATCH --nodes=1
#SBATCH -o slurmjob-%j.out-%N
#SBATCH -e slurmjob-%j.err-%N
#SBATCH --account=owner-guest
#SBATCH --partition=kingspeak-guest
#Set up whatever package we need to run with
module load somemodule
#set up the temporary directory
SCRDIR=/scratch/general/vast/$USER/$SLURM_JOB_ID
mkdir -p $SCRDIR
#copy over input files
cp file.input $SCRDIR/.
cd $SCRDIR
#Run the program with our input
myprogram < file.input > file.output
#Move files out of working directory and clean up
cp file.output $HOME/.
cd $HOME
rm -rf $SCRDIR
B
a
s
i
c
S
L
U
R
M
s
c
r
i
p
t
-
t
c
s
h
#!/bin/tcsh
#SBATCH --time=02:00:00
#SBATCH --nodes=1
#SBATCH -o slurmjob-%j.out-%N
#SBATCH -e slurmjob-%j.err-%N
#SBATCH --account=owner-guest
#SBATCH --partition=kingspeak-guest
#Set up whatever package we need to run with
module load somemodule
#set up the scratch directory
set SCRDIR /scratch/local/$USER/$SLURM_JOB_ID
mkdir -p $SCRDIR
#move input files into scratch directory
cp file.input $SCRDIR/.
cd $SCRDIR
#Run the program with our input
myprogram < file.input > file.output
#Move files out of working directory and clean up
cp file.output $HOME/.
cd $HOME
rm -rf $SCRDIR
R
u
n
n
i
n
g
i
n
t
e
r
a
c
t
i
v
e
b
a
t
c
h
j
o
b
s
•
An interactive command is launched through the
salloc command
salloc --time=1:00:00 –ntasks=2 --nodes=1 --
account=chpc --partition=kingspeak
•
Launching an interactive job automatically forwards
environment information, including X11 forwarding
allowing for the running of GUI based applications
O
p
e
n
O
n
D
e
m
a
n
d
i
s
a
n
o
t
h
e
r
o
p
t
i
o
n
t
o
s
t
a
r
t
i
n
t
e
r
a
c
t
i
v
e
s
e
s
s
i
o
n
s
(
p
r
e
s
e
n
t
a
t
i
o
n
–
T
h
u
r
s
d
a
y
,
F
e
b
r
u
a
r
y
9
,
2
0
2
3
)
--ntasks 2
P
a
r
a
l
l
e
l
E
x
e
c
u
t
i
o
n
•
MPI installations at CHPC are SLURM aware, so mpirun will
usually work without a machinefile (unless you are
manipulating the machinefile in your scripts)
•
If machinefile or host list needed, create the node list:
–
srun hostname | sort -u > nodefile.$SLURM_JOB_ID
–
srun hostname | sort > nodefile.$SLURM_JOB_ID
•
Alternatively, you can use the srun command instead, but you
need to compile with a more recently compiled MPI
•
Mileage may vary, and for different MPI distributions, srun or
mpirun may be preferred (check our slurm page on the CHPC
website for more info or email us)
S
l
u
r
m
f
o
r
u
s
e
o
f
G
P
U
N
o
d
e
s
•
GPU nodes are on lonepeak, kingspeak, notchpeak (and redwood in the PE)
•
Info on GPU nodes found at
https://chpc.utah.edu/documentation/guides/gpus-
accelerators.php
•
There are both general (open to all users) and owner GPU nodes (available via
owner-gpu-guest, with preemption, to all uses=rs
•
At this time, general GPU nodes are run without allocation
•
GPU partitions set up in a shared mode only as most codes do not yet make efficient
use of multiple GPUs so we have enabled node sharing
•
Must get added to the gpu accounts – request via
helpdesk@chpc.utah.edu
•
U
s
e
o
n
l
y
i
f
y
o
u
a
r
e
m
a
k
i
n
g
u
s
e
o
f
t
h
e
G
P
U
f
o
r
t
h
e
c
a
l
c
u
l
a
t
i
o
n
N
o
d
e
S
h
a
r
i
n
g
o
n
G
P
U
n
o
d
e
s
•
Need to specify number of CPU cores, amount of
memory, and number of GPU
•
Core hours used based on highest % requested
among cores, memory and GPUs
S
t
r
a
t
e
g
i
e
s
f
o
r
S
e
r
i
a
l
A
p
p
l
i
c
a
t
i
o
n
s
•
https://www.chpc.utah.edu/documentation/software/serial-jobs.php
•
When running serial applications (no MPI, no threads)
unless memory constrained, you should look to options to
bundle jobs together so using all cores on nodes
•
There are multiple ways to do so, including
–
srun --multi-prog
–
submit script
•
Also consider OpenScienceGrid (OSG) as an option
(especially if you have a large number of single core, short
jobs)
S
t
r
a
t
e
g
i
e
s
f
o
r
J
o
b
A
r
r
a
y
s
•
https://www.chpc.utah.edu/documentation/software/slurm.php#jobarr
•
Useful if you have many similar jobs when each use
all cores on a node or multiple nodes to run where
only difference is input file
•
sbatch --array=1-30%n myscript.sh – where n is
maximum number of jobs to run at same time
•
In script: use $SLURM_ARRAY_TASK_ID to specify
input file:
–
./myprogram input$SLURM_ARRAY_TASK_ID.dat
./myprogram input$SLURM_ARRAY_TASK_ID.dat
J
o
b
P
r
i
o
r
i
t
i
e
s
•
https://www.chpc.utah.edu/documentation/software/slurm.
php#priority
•
s
p
r
i
o
g
i
v
e
j
o
b
p
r
i
o
r
i
t
y
f
o
r
a
l
l
j
o
b
s
–
sprio –j JOBID for a given job
–
sprio –u UNID for all a given user’s jobs
•
Combination of three factors added to base priority
–
Time in queue
–
Fairshare
–
Job size
•
Only 5 jobs per user per qos will accrue priority based on
time on queue
C
h
e
c
k
i
n
g
J
o
b
P
e
r
f
o
r
m
a
n
c
e
•
With an active job
–
can ssh to node
•
Useful commands, top, ps, sar, atop
–
Also from interactive node can query job
•
/uufs/chpc.utah.edu/sys/installdir/pestat/pestat
–
Can query node status
•
scontrol show node notch024
•
After job complete -- XDMoD Supremm
–
Job level data available day after job ends
–
XDMoD sites
https://xdmod.chpc.utah.edu
and
https://pe-
xdmod.chpc.utah.edu
–
usage info:
https://www.chpc.utah.edu/documentation/software/xdmod.php
S
l
u
r
m
D
o
c
u
m
e
n
t
a
t
i
o
n
a
t
C
H
P
C
https://www.chpc.utah.edu/documentation/software/slurm.php
https://www.chpc.utah.edu/documentation/software/serial-jobs.php
https://www.chpc.utah.edu/documentation/software/node-sharing.php
https://www.chpc.utah.edu/usage/constraints/
https://www.chpc.utah.edu/documentation/guides/index.php#GenSlurm
O
t
h
e
r
g
o
o
d
d
o
c
u
m
e
n
t
a
t
i
o
n
s
o
u
r
c
e
s
http://slurm.schedmd.com/documentation.html
http://slurm.schedmd.com/pdfs/summary.pdf
http://www.schedmd.com/slurmdocs/rosetta.pdf
G
e
t
t
i
n
g
H
e
l
p
•
CHPC website
–
www.chpc.utah.edu
•
Getting started guide, cluster usage guides, software manual
pages, CHPC policies
•
Service Now Issue/Incident Tracking System
–
Email:
helpdesk@chpc.utah.edu
•
Help Desk: 405 INSCC, 581-6440 (9-6 M-F)
•
We use
chpc-hpc-users@lists.utah.edu
for sending messages
to users
Overview of SLURM commands, batch directives, environment variables, running interactive batch jobs, monitoring jobs, and getting additional information. Includes basic SLURM commands, useful aliases, and information on accounts and partitions for job submission.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
CENTER FOR HIGH PERFORMANCE COMPUTING Introduction to SLURM & SLURM batch scripts Anita Orendt Assistant Director Research Consulting & Faculty Engagement anita.orendt@utah.edu
CENTER FOR HIGH PERFORMANCE COMPUTING Overview of Talk Basic SLURM commands Accounts and Partitions SLURM batch directives SLURM Environment Variables SLURM Batch scripts Running an Interactive Batch job Monitoring Jobs Where to get more Information
CENTER FOR HIGH PERFORMANCE COMPUTING Basic SLURM commands sinfo - shows partition/node state sbatch <scriptname> - launches a batch script squeue - shows all jobs in the queue squeue -u <username> - shows only your jobs scancel <jobid> - cancels a job Notes: For sinfo, squeue can add M all to see all clusters using given slurm installation (notchpeak, kingspeak, lonepeak, ash) Can also add M cluster OR use full path /uufs/<cluster>.peaks/sys/pkg/slurm/std/bin/<command> to look at the queue, or submit or cancel jobs for a different cluster Redwood has own slurm setup, separate from others
CENTER FOR HIGH PERFORMANCE COMPUTING Some Useful Aliases Bash to add to .aliases file: alias si="sinfo -o \"%20P %5D %14F %8z %10m %10d %11l %16f %N\"" alias si2="sinfo -o \"%20P %5D %6t %8z %10m %10d %11l %16f %N\"" alias sq="squeue -o \"%8i %12j %4t %10u %20q %20a %10g %20P %10Q %5D %11l %11L %R\"" Tcsh to add to .aliases file: alias si 'sinfo -o "%20P %5D %14F %8z %10m %11l %16f %N"' alias si2 'sinfo -o "%20P %5D %6t %8z %10m %10d %11l %N"' alias sq 'squeue -o "%8i %12j %4t %10u %20q %20a %10g %20P %10Q %5D %11l %11L %R" Can add M to si and sq also You can find these on the CHPC Slurm page https://www.chpc.utah.edu/documentation/software/slurm.php#aliases
CENTER FOR HIGH PERFORMANCE COMPUTING Accounts & Partitions You need to specify an account and a partition to run jobs You can see a list of partitions using the sinfo command For general allocation usage the partition is the cluster name If no allocation (or out of allocation) use clustername-freecycle for partition Your account is typically your PI s name (e.g., if your PI is Baggins, use the "baggins" account) there are a few exceptions! Owner node accounts and partition have the same name PI last name with cluster abbreviation, e.g., baggins-kp, baggins-np, etc Owner nodes can be used as a guest using the "owner-guest" account and the cluster-guest partition Remember general nodes on notchpeak need allocation; general nodes on kingspeak and lonepeak are open to all users without allocation PE has its own allocation process
CENTER FOR HIGH PERFORMANCE COMPUTING More on Accounts & Partitions Allocations and node ownership status What resource(s) are available Unallocated general nodes Allocated general nodes in freecycle mode - not recommended Guest access on owner nodes No general allocation, no owner nodes Unallocated general nodes Allocated general nodes Guest access on owner nodes General allocation, no owner nodes Unallocated general nodes Allocated general nodes in freecycle mode - not recommended Group owned nodes Guest access on owner nodes of other groups Group owner nodes, no general allocation Unallocated general nodes Allocated general nodes Group owned nodes Guest access on owner nodes of other groups Group owner node, general allocation See https://www.chpc.utah.edu/documentation/guides/index.php#parts
CENTER FOR HIGH PERFORMANCE COMPUTING Query your allocation ~]$ myallocation You have a general allocation on kingspeak. Account: chpc, Partition: kingspeak You have a general allocation on kingspeak. Account: chpc, Partition: kingspeak-shared You can use preemptable mode on kingspeak. Account: owner-guest, Partition: kingspeak-guest You can use preemptable GPU mode on kingspeak. Account: owner-gpu-guest, Partition: kingspeak- gpu-guest You have a GPU allocation on kingspeak. Account: kingspeak-gpu, Partition: kingspeak-gpu You have a general allocation on notchpeak. Account: chpc, Partition: notchpeak You have a general allocation on notchpeak. Account: chpc, Partition: notchpeak-shared You can use preemptable GPU mode on notchpeak. Account: owner-gpu-guest, Partition: notchpeak- gpu-guest You can use preemptable mode on notchpeak. Account: owner-guest, Partition: notchpeak-guest You have a GPU allocation on notchpeak. Account: notchpeak-gpu, Partition: notchpeak-gpu You have a general allocation on lonepeak. Account: chpc, Partition: lonepeak You have a general allocation on lonepeak. Account: chpc, Partition: lonepeak-shared You can use preemptable mode on lonepeak. Account: owner-guest, Partition: lonepeak-guest You can use preemptable mode on ash. Account: smithp-guest, Partition: ash-guest
CENTER FOR HIGH PERFORMANCE COMPUTING Node Sharing Use the shared partition for a given set of nodes (using normal account for that partition) notchpeak* 2 2/0/0/2 2:16:2 768000 1700000 3-00:00:00 chpc,skl,c32,m768 notch[044-045] notchpeak* 32 32/0/0/32 4:16:2 255000 700000 3-00:00:00 chpc,rom,c64,m256 notch[172-203] notchpeak* 4 4/0/0/4 2:16:2 95000 1800000 3-00:00:00 chpc,skl,c32,m96 notch[005-008] notchpeak* 19 19/0/0/19 2:16:2 191000 1800000 3-00:00:00 chpc,skl,c32,m192 notch[009-018,035-043] notchpeak* 1 1/0/0/1 2:18:2 768000 7400000 3-00:00:00 chpc,skl,c36,m768 notch068 notchpeak* 7 7/0/0/7 2:20:2 191000 1800000 3-00:00:00 chpc,csl,c40,m192 notch[096-097,106-107,153-155] notchpeak-shared 2 2/0/0/2 2:16:2 768000 1700000 3-00:00:00 chpc,skl,c32,m768 notch[044-045] notchpeak-shared 32 32/0/0/32 4:16:2 255000 3700000 3-00:00:00 chpc,rom,c64,m256 notch[172-203] notchpeak-shared 4 4/0/0/4 2:16:2 95000 1800000 3-00:00:00 chpc,skl,c32,m96 notch[005-008] notchpeak-shared 19 19/0/0/19 2:16:2 191000 1800000 3-00:00:00 chpc,skl,c32,m192 notch[009-018,035-043] notchpeak-shared 1 1/0/0/1 2:18:2 768000 7400000 3-00:00:00 chpc,skl,c36,m768 notch068 notchpeak-shared 7 7/0/0/7 2:20:2 191000 1800000 3-00:00:00 chpc,csl,c40,m192 notch[096-097,106-107,153-155] In script: #SBATCH --partition=cluster-shared #SBATCH --ntasks=2 #SBATCH --mem=32G If there is no memory directive used the default is that 2G/core will be allocated to the job. Allocation usage of a shared job is based on the percentage of the cores and the memory used, whichever is higher https://www.chpc.utah.edu/documentation/software/node-sharing.php
CENTER FOR HIGH PERFORMANCE COMPUTING Owner/Owner-guest CHPC provides heat maps of usage of owner nodes by the owner over last two weeks https://www.chpc.utah.edu/usage /constraints/ Use information provided to target specific owner partitions with use of constraints (more later) and node feature list
CENTER FOR HIGH PERFORMANCE COMPUTING SLURM Batch Directives #SBATCH --time 1:00:00 wall time of a job (or -t) in hour:minute:second #SBATCH --partition=name partition to use (or -p) #SBATCH --account=name account to use (or -A) #SBATCH --nodes=2 number of nodes (or -N) #SBATCH --ntasks 32 total number of tasks (or -n) #SBATCH --mail-type=FAIL,BEGIN,END events on which to send email #SBATCH --mail-user=name@example.com email address to use #SBATCH -o slurm-%j.out-%N name for stdout; %j is job#, %N node #SBATCH -e slurm-%j.err-%N name for stderr; %j is job#, %N node #SBATCH --constraint C20 can use features given for nodes (or -C)
CENTER FOR HIGH PERFORMANCE COMPUTING SLURM Environment Variables Depends on SLURM Batch Directives used Can get them for a given set of directives by using the env command inside a script (or in a srun session). Some useful environment variables: $SLURM_JOB_ID $SLURM_SUBMIT_DIR $SLURM_NNODES $SLURM_NTASKS
CENTER FOR HIGH PERFORMANCE COMPUTING Basic SLURM script flow Set up the #SBATCH directives for the scheduler to request resources for job Set up the working environment by loading appropriate modules If necessary, add any additional libraries or programs to $PATH and $LD_LIBRARY_PATH, or set other environment needs Set up temporary/scratch directories if needed Switch to the working directory (often group/scratch) Run the program Copy over any results files needed Clean up any temporary files or directories 1. 2. 3. 4. 5. 6. 7. 8.
CENTER FOR HIGH PERFORMANCE COMPUTING Basic SLURM script - bash #!/bin/bash #SBATCH --time=02:00:00 #SBATCH --nodes=1 #SBATCH -o slurmjob-%j.out-%N #SBATCH -e slurmjob-%j.err-%N #SBATCH --account=owner-guest #SBATCH --partition=kingspeak-guest #Set up whatever package we need to run with module load somemodule #set up the temporary directory SCRDIR=/scratch/general/vast/$USER/$SLURM_JOB_ID mkdir -p $SCRDIR #copy over input files cp file.input $SCRDIR/. cd $SCRDIR #Run the program with our input myprogram < file.input > file.output #Move files out of working directory and clean up cp file.output $HOME/. cd $HOME rm -rf $SCRDIR
CENTER FOR HIGH PERFORMANCE COMPUTING Basic SLURM script - tcsh #!/bin/tcsh #SBATCH --time=02:00:00 #SBATCH --nodes=1 #SBATCH -o slurmjob-%j.out-%N #SBATCH -e slurmjob-%j.err-%N #SBATCH --account=owner-guest #SBATCH --partition=kingspeak-guest #Set up whatever package we need to run with module load somemodule #set up the scratch directory set SCRDIR /scratch/local/$USER/$SLURM_JOB_ID mkdir -p $SCRDIR #move input files into scratch directory cp file.input $SCRDIR/. cd $SCRDIR #Run the program with our input myprogram < file.input > file.output #Move files out of working directory and clean up cp file.output $HOME/. cd $HOME rm -rf $SCRDIR
--ntasks 2 CENTER FOR HIGH PERFORMANCE COMPUTING Running interactive batch jobs An interactive command is launched through the salloc command salloc --time=1:00:00 ntasks=2 --nodes=1 -- account=chpc --partition=kingspeak Launching an interactive job automatically forwards environment information, including X11 forwarding allowing for the running of GUI based applications OpenOnDemand is another option to start interactive sessions (presentation Thursday, February 9, 2023)
CENTER FOR HIGH PERFORMANCE COMPUTING Parallel Execution MPI installations at CHPC are SLURM aware, so mpirun will usually work without a machinefile (unless you are manipulating the machinefile in your scripts) If machinefile or host list needed, create the node list: srun hostname | sort -u > nodefile.$SLURM_JOB_ID srun hostname | sort > nodefile.$SLURM_JOB_ID Alternatively, you can use the srun command instead, but you need to compile with a more recently compiled MPI Mileage may vary, and for different MPI distributions, srun or mpirun may be preferred (check our slurm page on the CHPC website for more info or email us)
CENTER FOR HIGH PERFORMANCE COMPUTING Slurm for use of GPU Nodes GPU nodes are on lonepeak, kingspeak, notchpeak (and redwood in the PE) Info on GPU nodes found at https://chpc.utah.edu/documentation/guides/gpus- accelerators.php There are both general (open to all users) and owner GPU nodes (available via owner-gpu-guest, with preemption, to all uses=rs At this time, general GPU nodes are run without allocation GPU partitions set up in a shared mode only as most codes do not yet make efficient use of multiple GPUs so we have enabled node sharing Must get added to the gpu accounts request via helpdesk@chpc.utah.edu Use only if you are making use of the GPU for the calculation
CENTER FOR HIGH PERFORMANCE COMPUTING Node Sharing on GPU nodes Need to specify number of CPU cores, amount of memory, and number of GPU Core hours used based on highest % requested among cores, memory and GPUs Option Explanation request one p100 GPU (others types names are titanx, rtx3090, p100, v100, titanv, 1080ti, 2080ti, p40, t4, a40,a100) request 4 GB of RAM (default is 2GB/core if not specified) request all memory of the node; use this if you do not want to share the node as this will give you all the memory #SBATCH --tasks=1 requests 1 core #SBATCH --gres=gpu:p100:1 #SBATCH --mem=4G #SBATCH --mem=0
CENTER FOR HIGH PERFORMANCE COMPUTING Strategies for Serial Applications https://www.chpc.utah.edu/documentation/software/serial-jobs.php When running serial applications (no MPI, no threads) unless memory constrained, you should look to options to bundle jobs together so using all cores on nodes There are multiple ways to do so, including srun --multi-prog submit script Also consider OpenScienceGrid (OSG) as an option (especially if you have a large number of single core, short jobs)
./myprogram input$SLURM_ARRAY_TASK_ID.dat CENTER FOR HIGH PERFORMANCE COMPUTING Strategies for Job Arrays https://www.chpc.utah.edu/documentation/software/slurm.php#jobarr Useful if you have many similar jobs when each use all cores on a node or multiple nodes to run where only difference is input file sbatch --array=1-30%n myscript.sh where n is maximum number of jobs to run at same time In script: use $SLURM_ARRAY_TASK_ID to specify input file: ./myprogram input$SLURM_ARRAY_TASK_ID.dat
CENTER FOR HIGH PERFORMANCE COMPUTING Job Priorities https://www.chpc.utah.edu/documentation/software/slurm. php#priority sprio give job priority for all jobs sprio j JOBID for a given job sprio u UNID for all a given user s jobs Combination of three factors added to base priority Time in queue Fairshare Job size Only 5 jobs per user per qos will accrue priority based on time on queue
CENTER FOR HIGH PERFORMANCE COMPUTING Checking Job Performance With an active job can ssh to node Useful commands, top, ps, sar, atop Also from interactive node can query job /uufs/chpc.utah.edu/sys/installdir/pestat/pestat Can query node status scontrol show node notch024 After job complete -- XDMoD Supremm Job level data available day after job ends XDMoD sites https://xdmod.chpc.utah.edu and https://pe- xdmod.chpc.utah.edu usage info: https://www.chpc.utah.edu/documentation/software/xdmod.php
CENTER FOR HIGH PERFORMANCE COMPUTING Slurm Documentation at CHPC https://www.chpc.utah.edu/documentation/software/slurm.php https://www.chpc.utah.edu/documentation/software/serial-jobs.php https://www.chpc.utah.edu/documentation/software/node-sharing.php https://www.chpc.utah.edu/usage/constraints/ https://www.chpc.utah.edu/documentation/guides/index.php#GenSlurm Other good documentation sources http://slurm.schedmd.com/documentation.html http://slurm.schedmd.com/pdfs/summary.pdf http://www.schedmd.com/slurmdocs/rosetta.pdf
CENTER FOR HIGH PERFORMANCE COMPUTING Getting Help CHPC website www.chpc.utah.edu Getting started guide, cluster usage guides, software manual pages, CHPC policies Service Now Issue/Incident Tracking System Email: helpdesk@chpc.utah.edu Help Desk: 405 INSCC, 581-6440 (9-6 M-F) We use chpc-hpc-users@lists.utah.edu for sending messages to users
