Skip to main content

Installing Python Packages in Air-Gapped Research VMs

· 10 min read
Tera Insights Team
Tera Insights Team
tiCrypt Team

tiCrypt research VMs are air-gapped by design. They have no outbound internet access, which is fundamental to the security model but creates a practical problem: researchers need Python packages, and pip install needs a network. This article explains how the offline PyPI mirror solves that problem, why the isolation model works the way it does, and what the download strategy looks like under the hood.

For step-by-step instructions, see the admin setup guide and the researcher usage guide.

The Problem​

Python's package manager, pip, resolves and downloads packages from the public PyPI index over HTTPS. In an air-gapped VM, that network path does not exist. Without intervention, researchers are limited to whatever is baked into the VM image at build time.

That creates two scaling problems:

  1. Image bloat. Bundling packages into VM images means every new library request triggers an image rebuild, a slow process that affects all users.
  2. Domain diversity. A genomics researcher needs scanpy and anndata. A machine learning engineer needs PyTorch and transformers. A data analyst needs pandas and plotly. No single image can anticipate every team's requirements without becoming unmanageably large.

The mirror decouples package availability from image builds entirely. Packages live on the NFS share, not in the image, and researchers install what they need at runtime.

Architecture​

The offline PyPI mirror is a single self-contained shell script, pypi-mirror.sh, that operates in two modes depending on where it runs:

Download mode (run by an administrator on the NFS mount): the script downloads roughly 2,400 Python packages, their dependencies, and pre-compiled binary wheels from PyPI and the NVIDIA index directly into a pypi-packages/ directory on the NFS share.

Install mode (air-gapped researcher VM): resolves and installs packages from the local pypi-packages/ directory using pip's --no-index --find-links mechanism. No network traffic occurs. This works on both Linux and Windows Server VMs since pip's offline resolution is platform-agnostic. On Linux VMs, researchers can use the bash script directly; on Windows VMs, they configure pip with a pip.ini pointing at the share and use standard pip install commands.

The NFS share is the bridge between the internet and the air-gapped VMs. The script writes packages directly to it; researcher VMs mount it as a read-only share.

  ┌───────────────┐     ┌───────────────┐
  │  PyPI         │     │  NVIDIA PyPI  │
  │  (pypi.org)   │     │  (+ extras)   │
  └───────┬───────┘     └───────┬───────┘
          │    Internet         │
          └────────┬────────────┘
                   │ pypi-mirror.sh downloads
                   ▼
         ┌──────────────────┐
         │  NFS Share       │
         │  (read-only)     │
         │  pypi-packages/  │
         │  pip.conf        │
         │  manifest.json   │
         └────────┬─────────┘
                  │ reads
          ┌───────┴───────┐
          │               │
          ▼               ▼
  ┌───────────────┐ ┌───────────────┐
  │  Linux VM     │ │  Windows VM   │
  │  (air-gapped) │ │  (air-gapped) │
  │  pypi-mirror  │ │  pip install  │
  │  or pip       │ │  with pip.ini │
  └───────────────┘ └───────────────┘

The pip.conf Mechanism​

After every mirror, update, or add run, the script regenerates a pip.conf file alongside itself:

[global]
no-index = true
find-links = file:///mnt/nfs/pypi-mirror/pypi-packages

When a researcher copies this file to ~/.pip/pip.conf, every subsequent pip install command resolves from the local mirror directory and never attempts to reach the network. This means researchers can use standard pip workflows (pip install, pip install -r requirements.txt, pip install --upgrade) without any special flags or awareness of the mirror's internals. From their perspective, pip just works.

This is a deliberate design choice. Rather than wrapping pip in a custom tool or requiring researchers to learn new commands, the mirror slots into pip's existing configuration system. Any Python tutorial, Stack Overflow answer, or README that says "run pip install <package>" works as-is inside the VM.

For the full researcher workflow, see the Python Packages user guide.

Download Strategy​

Not every package publishes a pre-compiled binary for every platform. The script's download strategy accounts for this with a two-tier approach.

Tier 1: Binary Wheels​

For each package, the script first attempts to download pre-compiled wheels using:

pip download --only-binary=:all: \
  --platform manylinux2014_x86_64 --platform linux_x86_64 \
  --python-version 3.9 --python-version 3.10 --python-version 3.11 --python-version 3.12 \
  --abi cpXY --abi none

The manylinux2014_x86_64 platform tag covers the vast majority of Linux research VMs. Downloading wheels for Python 3.9 through 3.12 means researchers running any of those versions get a matching binary. The --abi none flag ensures pure-Python wheels (py3-none-any) are also captured.

Binary wheels are the preferred path because they install instantly (just unzip), require no compiler on the VM, and produce deterministic results regardless of what system libraries are available.

Tier 2: Source Distribution Fallback​

If no binary wheel exists for a package on any of the supported Python versions, the script falls back to downloading whatever pip can find, which is typically a source distribution (.tar.gz or .zip). This is the slower path: installing from source requires compiler tooling (gcc, python3-dev, make) on the VM, and the result depends on the VM's system libraries.

The fallback is intentionally permissive. Blocking source distributions entirely would leave gaps in the mirror for packages that only publish sdists (common in smaller scientific libraries). Downloading them ensures the mirror is as complete as possible, even if some packages require an extra build step on the VM.

NVIDIA Index​

CUDA and RAPIDS packages (cudf-cu12, cuml-cu12, cupy-cuda12x, and similar) are not hosted on the standard PyPI index. The script adds https://pypi.nvidia.com as an extra index URL so these packages resolve correctly without requiring administrators to configure anything.

Retry and Logging​

Every download attempt is retried up to 3 times with a short delay between attempts. This smooths over transient network hiccups during the download. Failures are logged to a timestamped file in logs/ so administrators can diagnose which packages did not download and why.

For setup instructions, see the admin guide.

Why Per-User Virtualenvs​

A shared VM may host multiple researchers. If packages installed system-wide, one user upgrading NumPy could break another user's analysis mid-run. The mirror avoids this by installing into per-user virtualenvs.

When a researcher runs ./pypi-mirror.sh install, the script:

  1. Checks for an already-activated virtualenv ($VIRTUAL_ENV).
  2. If none, looks for an existing venv at ./venv, ./.venv, or ~/venv.
  3. If none exist, creates a new one at ./venv in the current directory.

Packages install into that virtualenv, which is scoped to the individual user's home directory. This design gives each researcher:

  • Version independence. Different users can run different versions of the same package simultaneously.
  • No privilege escalation. No sudo is needed. Researchers manage their own environments.
  • Project-level isolation. A researcher can create separate venvs per project so that dependency sets stay independent.
  • Safe cleanup. Deleting a venv removes all its packages without affecting the system or other users. Recreating it from a requirements.txt takes seconds since everything resolves locally.

This is the same isolation model that Python developers use on internet-connected machines. The mirror makes it work identically in an offline environment.

The virtualenv approach also sidesteps a common operational headache: when system Python packages are managed by both the OS package manager (apt, dnf) and pip, they can conflict. By directing all researcher installs into venvs, the mirror never touches the system Python installation, which reduces the risk of breaking system tools that depend on specific package versions.

For practical tips on working with virtualenvs in tiCrypt VMs, see the Python Packages user guide.

What the Mirror Ships​

The curated package list covers 47 categories:

DomainExamples
Core numerical / dataNumPy, pandas, Polars, Dask, Arrow
Machine learningscikit-learn, XGBoost, LightGBM, CatBoost
Deep learningPyTorch, TensorFlow, JAX, Keras
LLM / generative AItransformers, langchain, openai, vllm
NLPspaCy, nltk, gensim, sentence-transformers
Genomics / bioinformaticsBiopython, scanpy, pysam, pyBigWig
Visualizationmatplotlib, plotly, seaborn, Bokeh
Geospatialgeopandas, rasterio, Fiona, shapely
GPU / RAPIDScudf, cuml, cupy, CUDA toolkit bindings
Statistics / Bayesianstatsmodels, PyMC, ArviZ, lifelines

Beyond the 643 curated packages, the mirror includes the top 1,744 PyPI packages by download count. These cover the transitive dependencies (packaging utilities, protocol buffers, HTTP clients, serialization libraries) that curated packages and researchers' own code frequently pull in. The combined catalog of roughly 2,400 packages means most pip install commands resolve on the first try without administrator intervention.

The curated list is organized by research domain rather than alphabetically, so administrators reviewing or extending it can quickly locate the category a new package belongs to. Administrators can also add any package on demand; see Adding Packages in the admin guide.

Manifest and Version Pinning​

After every mirror, update, or add run, the script generates a manifest.json file alongside pypi-packages/. This is a JSON inventory of every file in the mirror: package name, version, type (wheel or sdist), and file size. Administrators can use it to audit mirror contents, diff across update runs, or drive automated checks.

Because the mirror uses a flat directory structure, multiple versions of the same package coexist as separate files. When a researcher pins a version (pip install numpy==1.26.4), pip resolves against the local directory and picks the matching wheel. Older versions are never removed by update, so pinned installs remain stable even after the mirror is refreshed.

For version pinning examples, see the Python Packages user guide.

Concurrency and Safety​

The script acquires a lockfile (.pypi-mirror.lock) before writing to pypi-packages/. If a second mirror, update, or add invocation starts while one is already running, it exits immediately rather than corrupting the mirror. The lock uses flock where available, with a PID-file fallback for systems that do not support it.

This means administrators can safely schedule update via cron without worrying about overlapping runs, and researchers running install concurrently on different VMs never interfere with each other (install is read-only against the mirror directory).

Conda Interoperability​

Researchers who use conda for environment management can still install from the mirror. Conda environments ship their own pip, so the workflow is straightforward: create or activate a conda environment, then use pip install with the mirror's pip.conf or --find-links flag.

The mirror does not include conda-native packages (.conda / .tar.bz2 format from conda-forge, bioconda, or defaults channels). These are a fundamentally different package format served from different infrastructure. Conda channel mirrors are a much larger effort; conda-forge alone exceeds 1 TB. Most air-gapped deployments start with the pip mirror and add conda channels selectively if a specific package has no PyPI equivalent.

The practical recommendation: use conda to create and manage environments (Python version selection, activation, isolation), and use pip with the offline mirror to install packages into those environments. This gives researchers the organizational benefits of conda with access to the full mirror catalog.

Mixing conda and pip

When combining conda and pip in the same environment, install conda packages first, then pip packages. Reversing the order can cause conda's dependency solver to overwrite files that pip installed, leading to broken packages.

For step-by-step conda usage, see Using the Mirror with Conda in the user guide.

Further Reading​