Skip to main content

Submitting Jobs via Slurm

Last updated: August 12, 2026Latest Frontend Version: 2.17.9

Prerequisites

Access Level:

User, Sub-admin, Admin, Super-admin

Requirements

  • A tiCrypt VM configured as a Slurm submit host (managed by a system administrator)
  • Slurm installed and running in the VM image
  • Submission permissions on the relevant partition
  • Data and scripts staged within the tiCrypt project directory

Overview​

tiCrypt integrates with Slurm (Simple Linux Utility for Resource Management) for scheduling compute jobs on secure infrastructure. Jobs are submitted from within a tiCrypt virtual machine, either through the in-browser terminal or an RDP session.

Both sbatch (batch submission) and srun (direct execution, including interactive sessions) are supported.

Coming from a shared computing cluster?

tiCrypt runs two Slurm instances: a Global Slurm that schedules VM provisioning, and a Local Slurm inside your enclave that you interact with. The commands are identical to what you already know. See Shared Computing to tiCrypt for the full concept mapping.

info

The script and program names used in the examples below (analysis.sh, run_analysis.py, qc_cohort.R) are illustrative. The workload command may be any program callable from the shell: Python, R, compiled binaries, shell pipelines, Stata, MATLAB, etc.

Verify Slurm Availability​

Before submitting jobs, confirm that Slurm is available on the VM:

sinfo

If this command returns partition and node information, the VM is correctly configured for job submission.

Job Script Structure​

A job script has two parts: a block of #SBATCH directives followed by the shell commands to execute.

#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --output=logs/%x_%j.out
#SBATCH --error=logs/%x_%j.err
#SBATCH --time=01:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --partition=secure

echo "Running on $(hostname)"

Submit the script:

sbatch myjob.sh

%x expands to the job name and %j to the job ID. The logs/ directory must exist before submission.

Examples​

A minimal script that confirms successful submission and execution on a compute node.

#!/bin/bash
#SBATCH --job-name=hello
#SBATCH --output=hello_%j.out
#SBATCH --time=00:05:00
#SBATCH --mem=1G

echo "Hello from $(hostname)"
echo "Job ID: $SLURM_JOB_ID"
date

Job Arrays​

Job arrays execute the same script across multiple inputs in parallel. Each task receives its own index in SLURM_ARRAY_TASK_ID, which the script uses to select its slice of the work.

The most common pattern: one input per line in a manifest, with the task index selecting the line.

#!/bin/bash
#SBATCH --job-name=array_job
#SBATCH --array=1-50
#SBATCH --output=logs/%x_%A_%a.out
#SBATCH --error=logs/%x_%A_%a.err
#SBATCH --time=01:00:00
#SBATCH --cpus-per-task=2
#SBATCH --mem=8G
#SBATCH --partition=secure

INPUT=$(sed -n "${SLURM_ARRAY_TASK_ID}p" /project/myteam/inputs.txt)

python /project/myteam/scripts/process_one.py --input "${INPUT}"

Size the array to the manifest so the two never drift apart:

sbatch --array=1-$(wc -l < /project/myteam/inputs.txt) array_job.sh

%A represents the array job ID and %a the task index. The logs/ directory must exist before submission.

Manage arrays as a whole or per task:

squeue --me -t PENDING              # Tasks still queued
scancel <ARRAY_JOB_ID>              # Cancel every task
scancel <ARRAY_JOB_ID>_7            # Cancel only task 7
sacct -j <ARRAY_JOB_ID> --format=JobID,State,Elapsed,MaxRSS

Inline Submission with sbatch --wrap​

For single-command jobs, the script file can be omitted:

sbatch --wrap="echo \"Date: $(date)\"; echo \"Hostname: \$(hostname)\""

Unescaped expressions like $(date) are evaluated on the submit host before the job is queued. Escaped expressions like \$(hostname) are evaluated on the compute node at execution time.

Combine --wrap with resource directives:

sbatch \
    --job-name=quick_test \
    --time=00:30:00 \
    --cpus-per-task=4 \
    --mem=8G \
    --partition=secure \
    --wrap="python /project/myteam/scripts/sanity_check.py"

Direct Execution with srun​

srun runs a command directly under Slurm without a script file. It is suited to short commands, interactive sessions, and launching steps from within a job script.

Run a single command on a compute node:

srun --time=00:30:00 --cpus-per-task=4 --mem=8G --partition=secure \
    python /project/myteam/scripts/sanity_check.py

Open an interactive shell on a compute node:

srun --pty --time=01:00:00 --cpus-per-task=4 --mem=16G --partition=secure bash

The shell exits and the allocation is released when you type exit.

Use srun inside an sbatch script to launch job steps:

#!/bin/bash
#SBATCH --job-name=stepped_job
#SBATCH --time=02:00:00
#SBATCH --ntasks=4
#SBATCH --mem=16G
#SBATCH --partition=secure

srun --ntasks=1 python /project/myteam/scripts/step_a.py
srun --ntasks=1 python /project/myteam/scripts/step_b.py

GPU Workloads​

GPUs are requested with --gres=gpu:N, where N is the number of devices the job needs. Slurm sets CUDA_VISIBLE_DEVICES automatically, so frameworks such as PyTorch and TensorFlow detect the allocated devices without additional configuration.

info

GPU partition names and available device types are set per deployment. Run sinfo -o "%P %G" to list the partitions and the GPU resources each one offers before submitting.

#!/bin/bash
#SBATCH --job-name=train_model
#SBATCH --output=logs/%x_%j.out
#SBATCH --error=logs/%x_%j.err
#SBATCH --time=08:00:00
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --gres=gpu:1
#SBATCH --partition=gpu

module load python/3.11 cuda/12.4
source /project/myteam/venv/bin/activate

nvidia-smi

python /project/myteam/scripts/train.py \
    --data   /project/myteam/data/train/ \
    --output /project/myteam/models/

Confirm actual GPU utilization after a run completes so that future requests match what the workload uses:

sacct -j <JOBID> --format=JobID,JobName,Elapsed,AllocTRES%40,State

Resource Directives Reference​

The following flags apply to both sbatch (as #SBATCH directives or command-line options) and srun.

DirectiveDescription
--timeMaximum wall-clock time (HH:MM:SS). Jobs exceeding this limit are terminated.
--cpus-per-taskNumber of CPU cores allocated per task.
--ntasksNumber of parallel tasks. Set to 1 for non-MPI workloads.
--memTotal memory allocation (suffix G or M). Jobs exceeding this limit are terminated.
--partitionTarget partition. Available partitions are determined by the tiCrypt deployment.
--gresGeneric resources, used to request GPUs as gpu:N or gpu:<type>:N.
--arraySubmits a job array over an index range, optionally throttled with %N.

After a job completes, use sacct to review actual resource consumption and refine future requests.

Monitoring and Managing Jobs​

squeue --me                       # List your queued and running jobs
scontrol show job <JOBID>         # View detailed job information
scancel <JOBID>                   # Cancel a specific job
scancel --me                      # Cancel all your jobs
sacct -j <JOBID> --format=JobID,JobName,State,Elapsed,MaxRSS,ExitCode

Workflow Summary​

  1. Compose a job script with #SBATCH directives and the workload command.
  2. Submit the script with sbatch <script>.sh, or run a command directly with srun.
  3. Monitor active jobs with squeue --me.
  4. Review completed jobs with sacct.

The directive block stays the same across job types. Only the workload command changes.