SFTP 1.5.1
What is in this release
| Count | Summary | |
|---|---|---|
| New | 2 | Upload pipeline settings (upload_queue_depth, upload_workers), an optional pprof_listen profiling address |
| Change | 7 | A failed chunk fails the file, bounded memory per open file, background uploads, a shutdown that waits only for active transfers, contradictory session lifetimes rejected at startup, a write-ahead-logging database, a revised service unit |
| Fix | 4 | Chunk accounting at chunk boundaries, session cleanup when connections close together, leftover transfer state cleared at startup, the extend_by and max_lifespan session lifetimes applied |
New
Upload pipeline settings
Two settings in the [sftp] section of ticrypt-sftp.toml tune the upload pipeline. The defaults suit most deployments.
upload_queue_depth = 3 # finished chunks that may wait before writes block
upload_workers = 2 # chunks encrypted and uploaded concurrently
Raising them trades memory for throughput on high-latency links to tiCrypt.
Profiling address
An optional pprof_listen address exposes Go profiling endpoints for support investigations. It is disabled by default and should stay bound to loopback when enabled.
Changes
A failed chunk fails the file
Previously an upload failure was logged and forgotten while the client was told the file had been stored. Now a failed chunk fails the file: the SFTP client sees the error on close, the size is not recorded, and the transfer summary counts it as failed. A file that reports success is a file that arrived complete.
Memory is bounded
Chunk buffers are pooled and reused instead of allocated per chunk, and finished chunks move through a queue of fixed depth. When the queue is full the service stops accepting writes until an upload drains it, so the client slows down instead of the service growing.
Memory per open file is now bounded at roughly 120 MiB regardless of how large the file is or how fast the client sends. This is verified on every build: a test streams a synthetic file of arbitrary size through the real pipeline, checks every uploaded chunk byte-for-byte, and fails if buffer use grows with file size or if any buffer is left outstanding. It has been run at sizes from 64 MiB to 1 GiB with flat memory.
Writes no longer wait on the network
Each write used to block until a full 8 MiB chunk had been encrypted and pushed to tiCrypt, which stalled the client's request pipeline on every chunk boundary. Uploads now run in the background while the client keeps sending. On links where the round trip to tiCrypt is the limiting factor, this is the difference between one chunk per round trip and a continuously filled pipe.
Throughput depends on your network, host, and tiCrypt instance. Measure a representative transfer before and after upgrading rather than assuming a figure.
Shutdown waits only when there is something to wait for
Stopping or restarting the service now checks whether transfers are actually in progress.
At the moment of systemctl stop | Behavior |
|---|---|
| No active SFTP connections | Stops immediately |
| Transfers in progress | Waits up to 30 seconds, exits as soon as they finish |
| Still running at 30 seconds | Closes remaining connections, flushing buffered data and writing transfer summaries |
Both the command line and the log report which path was taken, with progress while waiting:
Received terminated: graceful shutdown started, 2 SFTP connection(s) in 1 transfer session(s)
still active. Shutting down in 30s, or sooner if they finish
Graceful shutdown: still waiting for 2 SFTP connection(s) in 1 session(s); 25s remaining
Graceful shutdown: all transfers finished after 12s. Shutdown complete
A restart during an upload no longer abandons transfer state.
Supporting changes
- Contradictory session lifetime settings are rejected at startup instead of being silently ignored.
- The local metadata store uses write-ahead logging, which removes most of the disk synchronization cost per file. Directory-heavy transfers with thousands of small files benefit most.
- The systemd unit now allows time for a graceful stop and raises the file descriptor limit for deployments with many parallel connections.
Fixes
Before this release, a large transfer could take the service down or leave state behind, and the user's next attempt failed with a permission or write-protection error that had nothing to do with their permissions. The failure had two halves, and they fed each other.
The upload pipeline could crash the service. Files are split into 8 MiB chunks, encrypted, and streamed into the recipient's inbox. The code that assembled those chunks miscounted bytes whenever a client's write request straddled a chunk boundary, which happens on every boundary with request sizes that do not divide evenly into 8 MiB. Miscounting led to a chunk being sent before it was complete, and to conditions the service could not recover from. Separately, every chunk allocated a fresh 8 MiB buffer and uploads ran without any limit on how many could be in flight, so a fast client on a good connection could drive memory into the hundreds of megabytes and be terminated by the host.
The service did not always clean up after itself. Transfer state is tracked per session in a local database and cleared when a user's last connection closes. Two connections closing at the same instant, the normal behavior of graphical SFTP clients, which open several in parallel, could each conclude the other was still active, and neither would clean up. A restart cleared memory but not the database, and because an inbox reuses the same SFTP username every time, the next transfer collided with the leftovers and was rejected.
Chunk accounting
Chunk accounting was rewritten. Every byte is placed by its position in the file, chunks are finalized only once their last byte has arrived, and a write that arrives for a region already sent is refused rather than silently dropped.
Sessions clean up exactly once
The end-of-session cleanup now makes its decision atomically, so parallel connections can no longer both defer to each other. Session state is scoped per transfer rather than per username. A transfer that ends badly no longer poisons the next one.
The service also no longer terminates on conditions it previously could not survive, including a failure to accept an incoming connection, the state that a busy host reaches after heavy transfer activity.
Leftover transfer state is cleared at startup
Any transfer state left behind by an earlier run is cleared when the service starts.
Web session lifetimes behave as documented
The extend_by and max_lifespan settings that govern tiCrypt user sessions on the inbox management API were parsed but never applied; sessions expired a fixed 30 minutes after sign-in regardless of activity. Sessions now extend while in use and stop at max_lifespan.
Upgrading
- Install the 1.5.1 package from your configured tiCrypt repository.
- Review
ticrypt-sftp.toml. No changes are required; the new[sftp]upload settings andpprof_listenare optional and default to previous behavior. - Restart the service. If a transfer is in progress it will be given the grace period described under Shutdown waits only when there is something to wait for.
If a previous version left transfer state behind, users may still have been seeing failures on repeat uploads to the same inbox. Version 1.5.1 clears that state at startup, so the first restart after upgrading resolves it. No manual database cleanup is needed.
Inbox behavior, credentials, and the sender-facing workflow are unchanged. External collaborators use the same SFTP clients and the same credentials page as before. See Inboxes for creating SFTP inboxes and sender instructions.