Skip to main content
tiCrypt 2.17.15

Submitting Jobs via Slurm

ResearcherRequires: User, Sub-admin, Admin, or Super-admin role
Permission requirements
  • A tiCrypt VM configured as a Slurm submit host (managed by a system administrator)
  • Slurm installed and running in the VM image
  • Submission permissions on the relevant partition
  • Data and scripts staged on an attached encrypted drive under /drives/

Overview​

tiCrypt integrates with Slurm (Simple Linux Utility for Resource Management) for scheduling compute jobs on secure infrastructure. Jobs are submitted from within a tiCrypt virtual machine, either through the in-browser terminal or an RDP session.

Both sbatch (batch submission) and srun (direct execution, including interactive sessions) are supported.

Write your results to a drive

Every example below reads and writes under /drives/, and yours should too. A job that writes anywhere else writes to the guest filesystem, which is discarded when the machine stops. For batch work this is not a theoretical risk: worker VMs are destroyed as soon as the job ends, so anything left outside /drives/ is gone before you can read it. See Where drives appear inside the VM.

info

tiCrypt VM images have no module system. There is no module load: the software you need is either baked into the image by your administrator, or installed into a virtual environment from the offline package mirror. See Python Packages.

Coming from a shared computing cluster?

tiCrypt runs two Slurm instances: a Global Slurm that schedules VM provisioning, and a Local Slurm inside your enclave that you interact with. The commands are identical to what you already know. See Shared Computing to tiCrypt for the full concept mapping.

info

The script and program names used in the examples below (analysis.sh, run_analysis.py, qc_cohort.R) are illustrative. The workload command may be any program callable from the shell: Python, R, compiled binaries, shell pipelines, Stata, MATLAB, etc. The examples also assume a drive named lab-data: tiCrypt links every drive under /drives/ by its name in title case, so it appears at /drives/Lab-Data on the VM and on the worker VMs that run your jobs.

Verify Slurm Availability​

Before submitting jobs, confirm that Slurm is available on the VM:

sinfo

If this command lists your partitions with a state of up, the VM is correctly configured for job submission. Expect the node count to be 0 while no jobs are running: tiCrypt creates a worker VM for each job and adds it to Slurm only when the job starts, so no compute nodes appear before you submit.

Job Script Structure​

A job script has two parts: a block of #SBATCH directives followed by the shell commands to execute.

#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --output=logs/%x_%j.out
#SBATCH --error=logs/%x_%j.err
#SBATCH --time=01:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --partition=secure

echo "Running on $(hostname)"

Submit the script:

sbatch myjob.sh

%x expands to the job name and %j to the job ID. tiCrypt replaces the name you set with --job-name by a generated name of the form job-<random>, so %x and squeue show that generated name. The logs/ directory must exist before submission.

Examples​

A minimal script that confirms successful submission and execution on a compute node.

#!/bin/bash
#SBATCH --job-name=hello
#SBATCH --output=hello_%j.out
#SBATCH --time=00:05:00
#SBATCH --mem=1G

echo "Hello from $(hostname)"
echo "Job ID: $SLURM_JOB_ID"
date

Job Arrays​

Job arrays execute the same script across multiple inputs in parallel. Each task receives its own index in SLURM_ARRAY_TASK_ID, which the script uses to select its slice of the work.

The most common pattern: one input per line in a manifest, with the task index selecting the line.

#!/bin/bash
#SBATCH --job-name=array_job
#SBATCH --array=1-50
#SBATCH --output=logs/%x_%A_%a.out
#SBATCH --error=logs/%x_%A_%a.err
#SBATCH --time=01:00:00
#SBATCH --cpus-per-task=2
#SBATCH --mem=8G
#SBATCH --partition=secure

INPUT=$(sed -n "${SLURM_ARRAY_TASK_ID}p" /drives/Lab-Data/inputs.txt)

python /drives/Lab-Data/scripts/process_one.py --input "${INPUT}"

Size the array to the manifest so the two never drift apart:

sbatch --array=1-$(wc -l < /drives/Lab-Data/inputs.txt) array_job.sh

%A represents the array job ID and %a the task index. The logs/ directory must exist before submission.

Manage arrays as a whole or per task:

squeue --me -t PENDING              # Tasks still queued
scancel <ARRAY_JOB_ID>              # Cancel every task
scancel <ARRAY_JOB_ID>_7            # Cancel only task 7
sacct -j <ARRAY_JOB_ID> --format=JobID,State,Elapsed,MaxRSS

Inline Submission with sbatch --wrap​

For single-command jobs, the script file can be omitted:

sbatch --wrap="echo \"Date: $(date)\"; echo \"Hostname: \$(hostname)\""

Unescaped expressions like $(date) are evaluated on the submit host before the job is queued. Escaped expressions like \$(hostname) are evaluated on the compute node at execution time.

Combine --wrap with resource directives:

sbatch \
    --job-name=quick_test \
    --time=00:30:00 \
    --cpus-per-task=4 \
    --mem=8G \
    --partition=secure \
    --wrap="python /drives/Lab-Data/scripts/sanity_check.py"

Direct Execution with srun​

srun runs a command directly under Slurm without a script file. It is suited to short commands, interactive sessions, and launching steps from within a job script.

Run a single command on a compute node:

srun --time=00:30:00 --cpus-per-task=4 --mem=8G --partition=secure \
    python /drives/Lab-Data/scripts/sanity_check.py

Open an interactive shell on a compute node:

srun --pty --time=01:00:00 --cpus-per-task=4 --mem=16G --partition=secure bash

The shell exits and the allocation is released when you type exit.

Use srun inside an sbatch script to launch job steps:

#!/bin/bash
#SBATCH --job-name=stepped_job
#SBATCH --time=02:00:00
#SBATCH --ntasks=4
#SBATCH --mem=16G
#SBATCH --partition=secure

srun --ntasks=1 python /drives/Lab-Data/scripts/step_a.py
srun --ntasks=1 python /drives/Lab-Data/scripts/step_b.py

GPU Workloads​

GPUs are requested with --gres=gpu:N, where N is the number of devices the job needs. Slurm sets CUDA_VISIBLE_DEVICES automatically, so frameworks such as PyTorch and TensorFlow detect the allocated devices without additional configuration.

info

GPU partition names and available device types are set per deployment. Run sinfo -s to list the partitions, but do not expect it to show GPU resources: worker nodes, and their GPUs, exist only while a job runs. Ask your administrator which partitions offer GPUs and which device types they provide.

#!/bin/bash
#SBATCH --job-name=train_model
#SBATCH --output=logs/%x_%j.out
#SBATCH --error=logs/%x_%j.err
#SBATCH --time=08:00:00
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --gres=gpu:1
#SBATCH --partition=gpu

source /drives/Lab-Data/venv/bin/activate

nvidia-smi

python /drives/Lab-Data/scripts/train.py \
    --data   /drives/Lab-Data/data/train/ \
    --output /drives/Lab-Data/models/

Confirm actual GPU utilization after a run completes so that future requests match what the workload uses:

sacct -j <JOBID> --format=JobID,JobName,Elapsed,AllocTRES%40,State

Resource Directives Reference​

The following flags apply to both sbatch (as #SBATCH directives or command-line options) and srun.

DirectiveDescription
--timeMaximum wall-clock time (HH:MM:SS). Jobs exceeding this limit are terminated.
--cpus-per-taskNumber of CPU cores allocated per task.
--ntasksNumber of parallel tasks. Set to 1 for non-MPI workloads.
--memTotal memory allocation (suffix G or M). Jobs exceeding this limit are terminated.
--partitionTarget partition. Available partitions are determined by the tiCrypt deployment.
--gresGeneric resources, used to request GPUs as gpu:N or gpu:<type>:N.
--arraySubmits a job array over an index range, optionally throttled with %N.

After a job completes, use sacct to review actual resource consumption and refine future requests.

Monitoring and Managing Jobs​

squeue --me                       # List your queued and running jobs
scontrol show job <JOBID>         # View detailed job information
scancel <JOBID>                   # Cancel a specific job
scancel --me                      # Cancel all your jobs
sacct -j <JOBID> --format=JobID,JobName,State,Elapsed,MaxRSS,ExitCode

Workflow Summary​

  1. Compose a job script with #SBATCH directives and the workload command.
  2. Submit the script with sbatch <script>.sh, or run a command directly with srun.
  3. Monitor active jobs with squeue --me.
  4. Review completed jobs with sacct.

The directive block stays the same across job types. Only the workload command changes.