Backend 3.14.1

Backend 3.14.1 is a storage and API release. The headline is that file garbage collection works again: files left behind by deleted users and removed directories have not been reclaimed on most deployments for a long time, and this release fixes the reason. Alongside it, the API gains multi-type tokens, ten read-only administration routes, and a capabilities endpoint that lets a client ask the backend what it supports rather than guess from a version number.

Nothing in this release reclaims storage on upgrade. The garbage collector ships disabled and in dry-run, and taking it live is a deliberate, staged decision documented in full below.

---

What is in this release

	Count	Summary
New	7	File garbage collection, multi-type API tokens, read-only administration routes, a capabilities route, immediate trash, automatic restore, a file manager startup check
Fix	4	Chunk deletion, project usage on modern MongoDB, a restore/purge race, stalled large-file scans
Change	3	Bulk delete response, garbage collector defaults, garbage collector log location

---

New

File garbage collection, rebuilt

When a user's home directory is removed, the files it contained stop being reachable but keep occupying storage. The collector was meant to reclaim them and did not: it had no reliable way to answer the question it depended on, does any directory still link to this file? Without that answer the sweep returned nothing, every pass, and orphaned files accumulated.

The collector now works from reference counts. Every directory entry that points at a file counts as one reference. A file with zero references belongs to no directory and nobody can reach it.

Collection happens in two passes, on separate schedules:

Pass	Runs every	What it does
Sweep	frequency (4h)	Counts references across all directories, finds files older than retention-period (60d) with zero references, and moves their records from file_infos to deleted_file_infos in batches of amount-grouped (100)
Purge	purge-frequency (24h)	Takes files that have been in the trash longer than deleted-grace-period (3d) and removes their keys, chunks, and record

The gap between the two passes is the safety margin. A file the sweep marks is not destroyed; it sits in deleted_file_infos for the grace period first. Every purge pass restores files that a directory links to again, whether or not dry-run is on, so a file that was marked and then relinked is put back rather than removed.

Every parameter is documented in the Garbage Collector configuration reference. The procedure for taking it live is below.

Multi-type API tokens

An API token used to carry a single type, which constrained it to the routes for that type. A Multi token now carries a set of types instead, any combination of Team, VM, Project, Drive and User, and may call the routes for every type in that set. One token can now do work that previously needed five.

This is what makes a whole-system snapshot possible from a single credential. See the REST API reference for the token model and System Snapshot for the script that uses it.

Read-only API routes for administration

Ten new read-only routes cover realms, bricks, external servers, security requirements and levels, permissions, project memberships, project and team usage, user certifications, and managed objects. They are read-only by design: they exist so that inventory, reporting, and compliance evidence can be collected without a credential that can change anything.

A capabilities route

Clients can now ask the backend which features it supports rather than infer it from a version string. The first three flags are remove-directory-entries, multi-type API tokens, and VM config drive keys. A frontend talking to a mixed estate can adapt per deployment instead of gating on a version comparison.

Immediate trash on bulk delete

Bulk directory entry deletion takes an immediate option that trashes newly unreferenced files straight away instead of waiting for the next sweep. This is the path to use when a user deletes a large directory and expects the quota to reflect it, rather than waiting up to frequency hours.

Automatic restore

A trashed file that is linked into a directory again is restored automatically on the next purge pass. This closes the window where a file could be marked, legitimately relinked, and then destroyed.

File manager startup check

The file manager now verifies its chunk storage path at startup, and reports missing chunks separately during purges rather than counting them as successful deletions. Both address the silent failure described under Verify your storage path.

---

Fixes

• Deleting a file removes its chunks from disk again. This had not happened since the storage service was removed, which means deployments have chunk files on disk whose records are already gone. See Know what this does not reclaim.
• Project usage aggregation no longer fails on MongoDB 3.6 and newer.
• A race between restoring and purging the same trashed file is resolved.
• Large file scans no longer stall after an idle connection between services.

---

Changes

• Bulk directory entry delete returns the directory together with the caller's updated file usage, so a client no longer needs a second call to refresh a quota display.
• The file garbage collector is disabled and in dry-run by default. Both switches must be changed deliberately: enabled = true creates the collector, and dry-run = false lets it act.
• Dry-run logs are written to /var/log/ticrypt/file-gc, configurable through log-directory, instead of /etc/ticrypt.

---

Upgrading

Upgrading reclaims nothing and changes no collector behaviour, because the collector ships off.

sudo dnf upgrade ticrypt-backend
sudo ticrypt-services.sh restart

Everything that follows is optional, and is the safe path to actually reclaiming the storage.

---

The garbage collector: going live safely

This procedure ends in permanent deletion

The purge removes keys, chunks, and records. There is no undo and no backup is taken first. Work through every step below, and do not shorten the dry-run period to save time.

Before you enable anything

Verify your storage path

This is the one check that matters most, and it is not optional.

ticrypt.filemanager.storage.path defaults to /tmp/ticrypt-storage, and the shipped file-manager configuration template does not mention the setting at all — so a deployment set up from the template is using the default.

If that path is wrong or not mounted, chunk deletion fails silently. A chunk that does not exist is treated as successfully deleted, so the purge reports complete success, removes the trash record, and leaves the chunk files on disk permanently, with nothing in any log to say so. The storage is never reclaimed and you have lost the record that would let you find it.

Confirm before you start:

grep -r "storage" /etc/ticrypt/ticrypt-file-manager.conf
df -h /your/storage/path        # is it mounted?
sudo -u ticrypt test -w /your/storage/path && echo writable

The file manager and the REST service must share this filesystem.

Expect the first live sweep to be large

Because the sweep has been returning nothing, every directory-less, unshared file older than your retention period is still on disk. The first live pass marks all of them, in batches of 100, and the first purge reclaims them a grace period later. Plan the first cycle for a window where you can watch it.

Know what this does not reclaim

Files deleted between December 2025 and 3.14.1 had their keys and records removed but their chunks left on disk. Those chunks appear in neither file_infos nor deleted_file_infos, so neither pass will ever find them. Reclaiming that backlog needs a separate reconciliation pass; contact support before attempting it.

---

Step 1: Run it in dry-run

Enable the collector while leaving dry-run on. Nothing is marked and nothing is purged; both passes write what they would have done to a log.

ticrypt.maintenance {
  garbage-collector {
    enabled  = true
    dry-run  = true

    retention-period     = 60d
    deleted-grace-period = 3d
    frequency            = 4h
    purge-frequency      = 24h
    amount-grouped       = 100

    scan-timeout  = 30m
    purge-timeout = 30m
    log-directory = "/var/log/ticrypt/file-gc"
  }
}

Restart ticrypt-maintenance and let it run for at least one full frequency cycle.

Step 2: Read the logs

The service journal

journalctl -u ticrypt-maintenance -f | grep -i "garbage\|purge"

Lines worth knowing:

Line	Meaning
FileGarbageCollector starting up (dryRun=true, ...)	Confirms the mode it came up in. Check this after every config change
Finished file garbage collection pass (dry run): would mark N files for deletion	The sweep ran. N is your candidate count
Finished deleted-file purge pass (dry run): would purge N files past their grace period	The purge ran
No directory entries found while N deletion candidates exist; skipping this garbage collection pass	A safety guard firing. The directory scan came back empty while candidates existed, so the pass refused to mark anything rather than mark everything. Investigate before continuing
Deleted-file purge slow or failed: ...	The purge exceeded purge-timeout. It keeps running; only the summary line is lost

The dry-run files

Both land in log-directory, timestamped in epoch milliseconds:

/var/log/ticrypt/file-gc/
├── file-garbage-collector-<timestamp>.log          # sweep candidates
└── file-garbage-collector-cleanup-<timestamp>.log  # purge candidates

The sweep log lists every file it would mark:

Removed 1284 files with no links to directories:
* 6f1a2c94-...
* 8b03df21-...

The cleanup log separates what would be destroyed from what would be saved:

Would purge 903 deleted files (14472 chunks) past their grace period:
* 6f1a2c94-... (16 chunks)
Would restore 4 deleted files still referenced by a directory:
* 2c77ba10-...

The sweep log says "Removed" during a dry run

The first line of the sweep log reads Removed N files... even when nothing was touched. Confirm the mode from the journal line, not from this wording.

Step 3: Spot-check the candidates

Do not skip this. Take ten file IDs from the sweep log and satisfy yourself that each one really is unreachable: no directory entry anywhere links to it, and no key belongs to another user. If any candidate turns out to be a live file, stop and open a support case — do not proceed to live.

Read the counts, too. If the sweep proposes to mark a number that surprises you for the size of your deployment, find out why before going further.

Run the dry run more than once before you go live

One clean dry run is not enough. Enable the collector in dry-run and let it complete several full frequency cycles, reading file-garbage-collector-<ts>.log after each one, before you ever set dry-run = false.

The reason is that the candidate set is not static. A file becomes a candidate the moment nothing links to it, so a single pass only shows you what was orphaned at that instant. Running repeatedly tells you two things one pass cannot:

• Is the list stable? The same IDs should keep appearing. A candidate set that changes substantially between passes means something is still moving files or directories, and marking them now risks catching live data.
• Is the volume what you expect? Compare counts across passes. A number that grows sharply, or that never matched your sense of the deployment's size, is a signal to stop and investigate rather than proceed.

Spot-check file IDs from each run, not only the first. Once dry-run = false, the sweep begins marking files for permanent deletion, and after deleted-grace-period elapses the purge removes their keys, chunks and records. That step cannot be undone.

Step 4: Go live

One flag governs both passes:

dry-run = false

Your settings stop being hypothetical

While dry-run = true, the parameters only decide what gets written to a log. The moment it is false, every one of them becomes an instruction the collector carries out. Read your own values before flipping the flag and be sure you agree with what each one now means.

With the shipped defaults:

enabled              = true
dry-run              = false
retention-period     = 60d
deleted-grace-period = 3d
frequency            = 4h
purge-frequency      = 24h
amount-grouped       = 100

that configuration means, in plain terms:

Setting	What it did in dry-run	What it does now
retention-period = 60d	Listed files older than 60 days with no directory links	Marks every such file for deletion
frequency = 4h	Wrote a candidate log every 4 hours	Marks a fresh batch every 4 hours, around the clock
amount-grouped = 100	Nothing	Marks in batches of 100 per pass
deleted-grace-period = 3d	Nothing	A marked file is destroyed 3 days after it is marked
purge-frequency = 24h	Logged what it would purge	Permanently removes keys, chunks and records once daily

So on a default configuration, a file that lost its last directory link 60 days ago is marked within 4 hours and is gone 3 days after that, with the purge running every 24 hours in between.

Worked examples of changing one value:

• retention-period = 7d reclaims space far sooner, and also catches files orphaned a week ago that a user may still be expecting to recover. A short retention on a first live run is the fastest way to delete something you wanted.
• deleted-grace-period = 14d is the setting to widen for a first live cycle. It does not change what gets marked, only how long you have to notice. Two weeks of marked-but-intact files is cheap insurance; three days is not much of a window if the first purge lands over a weekend.
• frequency = 24h slows the sweep to once a day, which makes the first live period far easier to watch than a pass every four hours.
• enabled = false stops everything. Files already marked stay in deleted_file_infos untouched and are restored if a directory links to them again.

A conservative first live configuration is therefore not the default one:

enabled              = true
dry-run              = false
retention-period     = 90d   # only long-orphaned files
deleted-grace-period = 14d   # two weeks to catch a mistake
frequency            = 24h   # one sweep a day, easier to watch
purge-frequency      = 24h
amount-grouped       = 100

There is no sweep-live, purge-dry-run state

dry-run controls the sweep and the purge together. Turning it off starts both.

What protects you is deleted-grace-period: for three days after the sweep marks a file, nothing is permanently removed, and any file that gets relinked is restored on the next purge pass. For the first live cycle, widen that window — set deleted-grace-period = 14d and you have two weeks to notice a mistake before anything is destroyed.

Restart, confirm dryRun=false in the journal, and watch deleted_file_infos fill as the sweep marks files. Nothing leaves disk yet.

Step 5: Through the first purge

Once the grace period elapses, the purge begins removing files permanently. Confirm:

• Finished deleted-file purge pass: purged N files past their grace period in the journal
• deleted_file_infos draining rather than growing without bound
• Reclaimed space on the storage filesystem

If anything looks wrong before the grace period ends, set enabled = false and restart. Files already marked stay in deleted_file_infos untouched.

---

Changes in the audit log

Bulk directory entry deletion now writes its own audit row, BulkDirectoryEntryRemove (BDER), recording the parent directory, the entries removed, whether the parent itself was deleted, and the outcome.

A file shown as deleted may still exist

An internal delete event is written when a file is trashed, and restores are not logged at all. An audit reader can therefore see a deletion for a file that the purge later restored because a directory still linked to it. Read deleted_file_infos alongside the audit trail when investigating a specific file.

---

Summary

Stage	Setting	What is at risk
Upgrade	enabled = false	Nothing
Observe	enabled = true, dry-run = true	Nothing. Logs only
Live sweep	dry-run = false	Nothing for deleted-grace-period
First purge	after the grace period	Files are permanently removed

Verify your storage path first, run in dry-run for a full cycle, spot-check the candidates, and widen the grace period for the first live cycle. The collector is deliberately slow to destroy anything, and every stage before the purge is reversible.

Questions about a specific deployment: contact our team.