Offline Python Package Mirror
For a technical explanation of the mirror's architecture, download strategy, and design reasoning, see the companion article. For researcher-facing usage instructions, see the Python Packages user guide.
Overview
tiCrypt research VMs are air-gapped: they have no internet access, by design. The offline PyPI mirror bridges that gap with a single self-contained shell script, pypi-mirror.sh, that runs directly on the NFS mount to download Python packages from the internet and make them available to researcher VMs.
The script is available at https://ticrypt.com/pypi-mirror.sh. It requires only python3 and pip; no other dependencies.
The mirror ships with roughly 2,400 packages: 643 curated research packages across 47 domains (data science, ML, deep learning, NLP, genomics, geospatial, visualization, and more) plus the top 1,744 PyPI packages by download count to cover transitive dependencies.
Setting Up the Mirror
Download the script directly onto the NFS mount:
curl -o /mnt/nfs/pypi-mirror/pypi-mirror.sh https://ticrypt.com/pypi-mirror.sh
chmod +x /mnt/nfs/pypi-mirror/pypi-mirror.sh
cd /mnt/nfs/pypi-mirror
Then run the mirror command (requires internet access to download packages):
./pypi-mirror.sh mirror
This downloads all ~2,400 packages (plus dependencies) into a pypi-packages/ directory next to the script. The first run creates the following layout:
pypi-mirror.sh the script itself
pypi-packages/ the mirror: downloaded wheels and sdists (flat directory)
manifest.json package inventory: every file, name, version, type, size
logs/ timestamped log file for every run
pip.conf generated pip config, ready for researchers to copy
custom-packages.txt admin-added packages (created by the 'add' command)
.pypi-mirror.lock lockfile guarding against concurrent runs
All package files (.whl, .tar.gz) are stored in a single flat pypi-packages/ directory. Multiple versions of the same package coexist as separate files (e.g., numpy-1.26.4-cp312-...whl alongside numpy-2.0.0-cp312-...whl). When update runs, the latest version is downloaded but older versions are not removed, so researchers can pin to either version.
manifest.json is regenerated after every mirror, update, or add run. It contains a JSON inventory of every file in the mirror with the package name, version, type (wheel or sdist), and file size. Use it to audit what is in the mirror, compare across update runs, or script automated checks.
A full mirror is roughly 60 to 100 GB, depending on which packages resolve to wheels versus larger source distributions. Make sure the NFS volume has headroom before the first run.
The script acquires .pypi-mirror.lock before writing to pypi-packages/, so a second mirror, update, or add invocation started while one is already running exits immediately instead of corrupting the mirror.
Keeping It Updated
Run update to refresh every package to its latest available version:
./pypi-mirror.sh update
update is functionally the same as re-running mirror: it re-downloads the full curated and custom package list, replacing outdated wheels with current ones.
Since researcher VMs never see the internet, package versions in the mirror only change when an admin refreshes it. Put update on a monthly or quarterly cron job so security patches and new releases reach VMs on a predictable cadence.
# Example: run on the 1st of every month at 2 AM
0 2 1 * * /mnt/nfs/pypi-mirror/pypi-mirror.sh update >> /var/log/pypi-mirror-cron.log 2>&1
Adding Packages
When a researcher needs a package that is not in the curated set, add it with:
./pypi-mirror.sh add <package> [package...]
This appends the package name to custom-packages.txt and downloads it into pypi-packages/ immediately. custom-packages.txt is a plain text file, one package per line, and it persists across update runs: once a package is added, every future mirror/update keeps it current along with the rest of the mirror.
Researchers cannot reach the internet to request packages themselves. The normal workflow is for a researcher to ask their admin, who runs add on the NFS mount. If a researcher has write access to the NFS share, they may also add lines to custom-packages.txt directly and ask the admin to run update (or add) to pull them down.
How It Works
For each package, the mirror (and update, add) commands try, in order:
- Binary wheels for Python 3.9, 3.10, 3.11, and 3.12 on the
manylinux2014_x86_64platform tag, usingpip download --only-binary=:all:. This covers the vast majority of packages and avoids compiling anything on researcher VMs. - A source distribution fallback for any package with no matching wheel on that platform for any of the supported Python versions. This is the slower path and can require build tools on the installing VM.
Downloads also point at the NVIDIA PyPI index (https://pypi.nvidia.com) as an extra index, so CUDA and RAPIDS packages (cudf-cu12, cuml-cu12, cupy-cuda12x, and similar) resolve correctly.
The script retries every download and install attempt up to 3 times, with a short delay between attempts, before recording it as a failure. This smooths over transient network hiccups without requiring a full re-run.
Finally, mirror, add, and update all regenerate pip.conf:
[global]
no-index = true
find-links = file:///mnt/nfs/pypi-mirror/pypi-packages
This pip.conf is what researchers on Linux VMs use to point pip at the local mirror. Researchers on Windows Server VMs need a pip.ini file at %APPDATA%\pip\pip.ini with the same content, adjusted for the Windows mount path (e.g., find-links = Z:\pypi-mirror\pypi-packages). See the Python Packages user guide for the researcher-facing setup steps on both platforms.
Built-in Reference
The script includes a built-in reference guide covering both admin and researcher workflows. Run:
./pypi-mirror.sh readme
This prints a self-contained quick-reference to the terminal, including setup steps, all install modes, tips, and conda usage. It is useful for onboarding new administrators or researchers who do not have access to the web documentation.
Troubleshooting
"Another instance of pypi-mirror.sh appears to be running" but nothing is actually running.
This means a run that did not exit cleanly (for example, a killed process) left .pypi-mirror.lock behind. Confirm no mirror, update, or add is genuinely in progress, then remove the stale lockfile:
rm /mnt/nfs/pypi-mirror/.pypi-mirror.lock
Mirror takes too long to run.
The first mirror run downloads all ~2,400 packages and their dependencies. Subsequent update runs are faster since pip caches resolved metadata. If a run fails partway through, re-run the same command; the script picks up where it left off for packages already in pypi-packages/.
A package fails to download.
The script retries each package up to 3 times. If it still fails, the package name is logged. Check logs/ for the timestamped log file. Common causes: the package does not publish wheels for manylinux2014_x86_64, or it is only available on a private index the script does not know about. For CUDA/RAPIDS packages, verify the NVIDIA index is reachable from the NFS mount server.
Researchers report "No matching distribution found" for a package that is in the mirror.
The mirror may only contain a source distribution for that package. On Linux VMs, the researcher needs compiler and header packages (gcc, python3-dev or python3-devel, make) to build it. On Windows VMs, the equivalent is the Visual C++ Build Tools. Either install build tools on the VM image, or find a prebuilt wheel from another source and place it in pypi-packages/ manually.
Windows VM researchers cannot find the mirror.
Confirm the NFS share is mounted and accessible from the Windows VM. Windows Server requires the NFS Client feature to be installed (Install-WindowsFeature NFS-Client in PowerShell). Once mounted (e.g., mount \\nfs-server\pypi-mirror Z:), researchers use pip directly with --find-links Z:\pypi-mirror\pypi-packages or by configuring pip.ini. See the user guide for Windows pip.ini setup.