Skip to main content

Filesystem and tiCrypt

· 7 min read
Alin Dobra

tiCrypt Vault Storage

The tiCrypt Vault offers a file system-like facility that allows files and directories to be created and used. All the metadata, such as file properties, directory entries, access information, decryption keys, are stored in the MongoDB database used by the tiCrypt-file storage service.

In tiCrypt Vault, the file content is broken into 8MB chunks, each encrypted independently using the file key. The size of such chunks is 8MB+64Bytes on disk (unless they are the last, incomplete chunk ). The extra 64Bytes contain the IV (initialization vector for AES encryption). For each file, the chunks are numbered from 0 onwards. Based on the file ID, the chunks are stored in a directory structure, visible only to the tiCrypt backend (preferably not to the VM hosts). The location of the storage can be configured in the configuration file of the tiCrypt-storage service.

tiCrypt is not opinionated on the file system or integration with other systems for this storage but, for compliance reasons, it is recommended that the access is restricted to only the tiCrypt backend.

Without the decryption keys that only users can recover, the content of the chunk files is completely indecipherable. It is safe to back up these files using any method (including non-encrypted backup, cloud, etc). The strong encryption coupled with the unavailability of the key to even administrators ensures that this content can be replicated outside the secure environment from a compliance point of view.unavailability of the key to even administrators, ensures that from a compliance point of view, this content can be replicated outside the secure environment.

tiCrypt Encrypted Drives in Libvirt

tiCrypt virtual machines use a boot disk image that is not encrypted and one or more encrypted drives. Both disk images and encrypted drives are stored as files in the underlying distributed file system available to all Libvirt host servers. The specific mechanism uses the notion of Libvirt disk pools. Independent disk pools can be defined for disk images, encrypted drives, drive snapshots (for backup), ISOs, etc. Each pool is located in a different directory within the distributed file system.

Libvirt (and tiCrypt, by extension) is agnostic to the choice of the file system where various disk pools are defined. A good practice is to place the various disk pools on a distributed file system visible (preferably in the same location) and to mount the file system on all VM hosts. Any file system can be used, including NFS, BeeGFS, Luster, etc.

As part of the virtualization mechanism, Libvirt makes the files corresponding to drives, stored on the host file system, appear as devices to the OS running within the VM. Any writes to the virtual device get translated into changes of the underlying file. The situation is somewhat more complex when snapshots are used since multiple files on the disk will form the virtual device.

Encrypted Drive Creation

Upon drive creation, tiCrypt instructs Libvirt to create a drive. This results in a file being created in the underlying file system in the corresponding drive pool. Two types of drives are supported: raw, with extension .raw, and QCOW2, with extension .qcow2.

For a raw drive, a file as large as the indicated drive size gets created. The file content is initialized with 0s (corresponding to a blank drive). Writes in the virtual drive result in writes in the corresponding file at exactly the same position (e.g., if block 10244 of the virtual drive is written, block 10244 of the raw file gets changed as well)

For a QCOW2 drive, only changed blocks get written; the file format is quite complex and supports advanced features like copy-on-write. The initial file size is small (low megabytes) when the drive is new; the file grows in size as more data is written to the disk.

The qemu-img tool can be used to convert between two formats. Usually, tiCrypt takes care of setting the drives up without the need for this tool.

A newly created tiCrypt disk is blank. Absolutely no formating of the drive or any other preparation has been performed. The drive will be formated the first time it is attached to a tiCrypt virtual machine. The main reasons for this are:

  • The encryption/decryption key for the encrypted drive is kept secret from the infrastructure. This includes the tiCrypt backend, Libvirt, and underlying VM host.
  • The choice of the file system to use is delegated to the underlying operating system and tiCrypt VM Controller. Libvirt is not aware, nor does it needs to know, of the actual file system on the drive. For Linux formated drives, by inspecting the files backing up the drives, there is no way to tell even if the drive is formatted at all, let alone any information on content or type of file system.

Encrypted Drive Formatting

As far as Libvirt is concerned, only low-level disk reads and writes exist. Whatever operation the operating system is performing gets translated into reading/write operations on the virtual disk; in turn, this will result in reading/write operations for the underlying file in the disk pool.

In Windows, a normal NTFS file system is crated but immediately (before the drive is made available to the user), BitLocker is turned on. This ensures that all files created subsequently are encrypted. BitLocker uses so-called "full volume encryption," i.e., all the new data will be encrypted, including the metadata. An external tool scanning the backing file can determine that the drive is an NTFS formatted drive and read all non-encrypted content. Sine tiCrypt turns on encryption immediately, very limited information is visible.

In Linux, the LUKS full disk encryption mechanism is used. It essentially places an encryption block layer between the raw drive (virtual drive in this case) and the file system (usually EXT4). This way, absolutely all information on the disk is encrypted. An external tool can only tell which disk blocks have been written to (are non-zero) but can derive no information on the rest of the content

tiCrypt Non-secure Drives

Two types of non-secure drives are supported in tiCrypt: ISOs and read-only NFS shares.

Attaching ISOs

ISOs are made available using read-only CD-ROM devices. As such, they are always safe to mount in a secure tiCrypt VM. Both Linux and Windows can readily mount such ISOs and make them available as "drives."

ISOs are particularly useful if the NFS shares described below are not used. For example, Python or R packages could be made available as ISOs so that various VMs can install the required packages locally.

Attaching NFS file systems

By allowing, through firewall rules, access to a local NFS server, various tiCrypt VMs can mount a common file system for the purpose of accessing public (non-secure) data, packages, software, etc.

From a security point of view, the NFS server should export the data as read-only. The tiCrypt secure VMs should never be allowed to mount a read-write NFS share since data can be exfiltrated, thus defeating the great effort put into tiCrypt protections against data exfiltration. This will most surely make the tiCrypt deployment non-compliant.

A further restriction is related to the location of the NFS server. The NFS file system must be under full control of the system administrators of tiCrypt. It has to be part of the tiCrypt system envelope; for example, one of the server's part of the tiCrypt infrastructure can take this role. This restriction is due to compliance complications: the security envelope extends to all parts of the system; a remote NFS server becomes part of the secure environment and is subject to all the security restrictions.

A practical recommendation is to create a local NFS server inside the security envelope and regularly sync content from an external NFS server. System administrators responsible for the operations need to be identified in the security plan (compliance requirement).