Skip to main content

tiCrypt Slurm Integration Planning

· 7 min read
Alin Dobra

Adding batch-processing capabilities to tiCrypt is one of the most requested new features. It will allow large computational jobs to be executed by the secure environment provided by tiCrypt with full cryptographic isolation and security, achieving batch processing in a fully compliant CMMC/NIST environment.

A natural solution is integrating tiCrypt with Slurm, the most popular batch-processing system. This document provides a technical discussion of the integration and security challenges.

Slurm Overview​

Slurm is a batch-processing system that allows users to submit jobs to a cluster of machines. The jobs are queued and executed when resources become available. Slurm provides a rich set of features, including:

  • sophisticated job scheduling
  • resource management
  • job accounting and reporting (including billing)
  • job execution, most notably for MPI jobs
  • job monitoring and control

For secure computation, especially when covered by CUI restrictions, Slurm is a poor choice. While building somewhat secure systems around Slurm is possible, it is difficult and often results in significant performance degradation. The main difficulty to overcome is that Slurm is designed to run jobs on the same machine where the Slurm controller is running. This does not protect against malicious code running on the same machine as the Slurm controller. Moreover, Slurm cannot isolate jobs or infrastructure from each other.

Ideal tiCrypt-Slurm Integration​

The ideal integration of tiCrypt with Slurm would provide the following features:

  • Isolation of jobs from each other. This is the most critical feature. It means that jobs cannot interfere with each other,
  • Isolation of jobs from the infrastructure. This is also very important. Ideally, Slurm should not be aware of the code that is being executed, nor does it have access to the data processed.
  • Integration with tiCrypt. The integration should be as seamless as possible. In particular, data encrypted by tiCrypt should be seamlessly integrated.
  • Minimal performance degradation. The integration should not significantly degrade the performance of Slurm. Ideally, it should not degrade the performance at all.
  • Keep the excellent Slurm capabilities. tiCrypt should rely on Slurms scheduling, resource management, job accounting and reporting, job monitoring, and control capabilities.

Architecture of the solution​

The above goals seem difficult to achieve because most of Slurm's capabilities must be retained, but the security must be "fixed". The key idea is to separate Slurm functionality into two parts:

  1. Global Slurm Scheduler. This is part of Slurm that is responsible for scheduling jobs, managing resources, accounting, and reporting. This component knows who executes the jobs, what resources they require, tracks the jobs, etc. This component is unaware of the code being executed nor has access to the data processed. This Slurm instance will run globally, outside tiCrypt, and interact with tiCrypt Backend through Slurm REST API.

  2. Local Slurm Executor. This is the part of Slurm that is responsible for executing the jobs. This component is aware of the code that is being executed and has access to the data processed. It can also provide Slurm with advanced execution capabilities, such as MPI. This Slurm instance will run locally, inside tiCrypt-managed Virtual Machines, and interact only with tiCrypt VM Controller. Each CUI project will have its own local Slurm Executor, managed by the tiCrypt VM Controller.

The Global Slurm Scheduler will not be aware of the Local Slurm Executors, and viceversa. The interaction between the two will be through the tiCrypt Backend and tiCrypt VM Controller. The tiCrypt VM Controller will hide most details from the tiCrypt backend (such as the code being executed, the data processed, etc.) and only provide information on the resources needed. This ensures that the Global Slurm Scheduler is not aware of the code being executed nor has access to the data processed. The tiCrypt backend will hide global details such as what other jobs run in the system, who is running them, etc. This will ensure the fact that the Local Slurm Executor is not aware of what else is running in the system, who is running it, etc.

The main mechanism used by tiCrypt to "trick" Slurm into running, as described above, is extensive use of the Slurms Plugging Architecture and re-write the jobs and statistics reporting mechanism. The following two sections describe the specific mechanisms to achieve the above goals.

Global Slurm Scheduler​

The Global Slurm Scheduler will be a full-fledged Slurm instance, running outside of tiCrypt but side-by-side with tiCrypt backend. Specifically, slurmctld, slurmdbd and slurmrestd will run on the same machine as the tiCrypt backend. It will be configured separately from tiCrypt and can use any of the Slurm features, most importantly, various plugins. Slurm will be configured only to allow the tiCrypt backend to submit jobs; specifically, the Slurm API will be guarded against any other access.

For each of the VM Hosts used by tiCrypt for batch processing, a Slurm node will be configured. Specifically, slurmd will run on each VM Host. tiCrypt backend will feed "fake" jobs to Slurm to simulate the actual execution (see below). Special programs that do nothing except coordinate with tiCrypt Backend will be submitted to Slurm. This technique is similar to PCOCC system. Custom LUA plugins will be used to intercept the job execution (quirk in Slurm that this is the only way to block job execution) and to "adjust" the job statistics based on actual execution (see below).

tiCrypt Controlling VM​

For each CUI project, tiCrypt will create an interactive VM (CVM) that will be used to run the jobs for that project. The project members will fully control this VM and will be used to manage the Local Slurm Executor, provision all the security, and interact with tiCrypt Backend. This mechanism is similar to the management of tiCrypt Interactive Clusters. Specifically, the CVM will:

  • Interact with tiCrypt users using the exact mechanism as the regular tiCrypt interactive VMs.
  • Provide the distributed file system used by the batch execution. This will use the exact mechanism as the tiCrypt Interactive Clusters.
  • Provide secure communication between the CVM and the worker VMs using automatically provisioned VPN (using StrongSwan).
  • Manage the Local Slurm Executor and local job submission
  • Ask tiCrypt Backend for VMs to be provisioned for batch processing. This is based on submitted Slurm jobs.
  • Inform Local Slurm Executor when resources are available and jobs can be executed.
  • Decommission the VMs when the batch processing is done.

Users with access to the CVM will be able to submit jobs to the Local Slurm Executor using sbatch and srun commands.

Local Slurm Executor​

The Local Slurm Executor will be a full-fledged Slurm instance, running inside tiCrypt-managed Virtual Machines. Specifically, slurmctld ( and if strictly necessary, 'slurmdbd) will run on the CVM, and slurmd` will run on each VM part of the batch job (managed by local VM Controllers coordinated via the CVM). The provisioning and execution will be controlled by the tiCrypt CVM. Via Slurm command line tools or tiCrypt Frontend integration, the users can see the jobs' status, cancel them, etc.

To provide integration with the rest of tiCrypt infrastructure and Global Slurm Schedler, LUA plugins and other tiCrypt mechanisms will intercept the job submission, execution, and statistics reporting.

Challenges and Solutions​

The above plan might sound complicated, but we think the complexity is manageable. The main challenges are:

  • Learning Slulrm. While the Tera Insights team has no experience running Slurm at scale, we have extensive experience dealing with complex systems and "coding around" their limitations and quirks.
  • Using LUA plugins. The Slurm documentation on plugins is not particularly extensive, but we can draw much inspiration from the PCOCC system implementation.
  • Adding new features to tiCrypt. The tiCrypt team has extensive experience in adding new features to tiCrypt. We have a well-established process for adding new features, including design, and implementation.