Skip to main content

· 19 min read
Betuel Gag

We will explore what administrators can do to manage tiCrypt for their teams. This practical guide should help administrators with the following:

  • How to practically make the best use of the system
  • How to help their team effectively
  • How to innovate in tiCrypt and add their blend to it

This document is not intended to replace tiCrypt documentation and to provide exhaustive information on tiCrypt features. For technical reference, please refer to tiCrypt documentation. The sister document addressing the best practices for project management may be found in administrator roles within tiCrypt.

The Principle of Minimal Privilege (Zero-Trust)

The term 'Zero-trust' is broadly misunderstood and often interpreted as a security principle for marketing purposes.

Zero-trust should be called the Principle of Minimal Privilege.

'Minimal' means never wanting to give more permission than is needed to any user in the system, including access to the data and other users' data. Zero-trust is a minimal privilege and preferably cryptographically enforced.

Talking about the minimal privilege principle is consequential because it is a complex problem. The government is skilled at putting zero-trust documents together but needs to be more skilled at doing things in practice. What needs to be addressed is enforcing the minimal privilege into a fully functional cryptographically enforced system, a challenge solved by tiCrypt projects.

To the majority, zero-trust is a mechanism that allows users to take a specific action based on a set of rules written by the administrator in the beginning. However, the constant problem with this methodology is the following: when a hacker comes in, he won't use the admin rules but will instead find a way to circumvent the whole system, so now the admin's zero-trust is worthless.

Imagine the following scenario.

A hacker attempts a social engineering attack ( or admin impersonation) where he asks for a "password reset for his account." This hacking attempt may work with large organizations of thousands of people; however, it is a handicapped attempt for tiCrypt.

In a tiCrypt scenario, the employee at the end of the line would not be able to give any details to an admin-impersonated hacker due to the minimal privilege principle.

Since they have no access to the data themselves, employees would have to forward the hacker to the project leader of the specific team, who has access to the particular data he requested in the first place. In a small team, the project leader has the group keys and knows all the team members because they work closely together. Hence they would immediately recognize the social engineering attack and calmly expose the hacker.

Furnishing Compliance

Compliance is complex, but on the other hand, it allows the flow of research grants and increases research expenditure.

To furnish compliance, admins should use projects that include security levels & requirements allowing access to research data based on the user certifications.

Admins can leverage the user certifications to external organizations who offer relevant research materials to successfully carry out a project in exchange for complying with their requirements. The operation opens many doors to high-quality data research entities.

We recommend adding an expiry date to the user certificates based on good practice to enforce compliance.

Admins should keep security requirements as simple as possible to avoid unnecessary questions from many users. At the same time, security levels are better off when using one word or simple abbreviations (i.e., HIPAA).

When considering compliance per project, admins may view the initial constraints and restrictions. In contrast with the project requirements, they can compile a set of rules to help every member comply before project deployment.

Data Back-up Management

There are two practices for backup: never-back-up data & always-back-up data.

Never-back-up data may be used for general repetitive data representing a storage liability to the organization. This practice may be used for long-term cleaning storage purposes.

Always-back-up data may be used for collectible data assets resulting from various research projects; files that have to do with system functionality; parameters of compliance; and anything that may have a significant impact.

note

Oftentimes administrators are requested by the users to keep all their data backed-up. They should not do that if they want to avoid a larger storage problem.

Suppose a University has attributed 10TB drives for researchers to produce significant data stocks; it will develop normally until the admins decide to back up the data.

Back-uping the data is more expensive than creating it because it requires the drives to keep the entire history of files. The university has an add-on that eventually becomes ten times bigger than 10TB, ever-growing daily to become the most expensive part of the system. This issue leads to choosing what gets backed up and where the backup takes place.

In good practice, administrators should be highly selective with the backup operations; they should constantly decentralize it.

When backup data is decentralized, it allows control over backup sections so that admins decide what remains or goes out of the system.

The only scenario when administrators can enforce data backup is when they force users to commit large amounts of financial resources for the service. Some universities run into millions of dollars in costs because of data-back-up management.

Another important aspect of backup is data access management.

In a hypothetical example, a user has two levels of data to work with: Federal government-level & Federal state-level data.

The management rules are 'never mix the data from the two levels' and 'do not peek at other federal government data files.

A user is naturally tempted to see other federal data files unless the system enforces this operation. tiCrypt enforced compliance mechanism allows the user to see specific parts of data required for the project while keeping other data separate and inaccessible. Users cannot mix data unless expressly permitted by the group owner or administrator.

To achieve this in practice, administrators should tag the projects with two different security levels allowing the user to access only specific data sets needed for the project. Administrators can use groups to stop a member from adding files to a local project, avoiding the mix of data altogether.

Escrow Mechanism

tiCrypt understands users of any technical level, from simple research activities to complex operations.

Login passwords are bound to be in a private password manager or offline in a safe place. The users do not have to memorize any keys or extended characters except their user's email address and a memorable password.

info

Users should be advised to never edit any characters in their private key. A single edit can deactivate their key resulting in an unpleasant escrow situation.

To keep the private key safe and avoid escrow protocols, administrators should do the following:

  • Put the private key in a location they can easily find it
  • Always make at least one copy to their private key and keep it somewhere safe, either offline or in their local machine

By doing this, administrators will be able to log in if they forget their previous private key location and give access to new members to join their group.

To help administrators understand the principle behind the escrow mechanism, we showcase the following box analogy. All private keys in tiCrypt are encrypted; they exist in closed virtual boxes that require a cryptographic key pair. If hackers would like to crack a ' box' where the user has its private key, it would take all the computing resource power of humanity running at the age of the universe. The exact process appears when encrypting files and escrow keys.

Since there is no ' forgot my password' button in tiCrypt due to strong security, the system uses escrow.

Escrow is a separation of duties; it works only if multiple escrow users do it. The escrow key may be split into three parts where all escrow users will share their piece of the key with the fourth user who needs to escrow their account.

The best practice to avoid bad key management is to have three escrow users who are not regular users, system admins, or audit members.

note

An escrow user cannot log in to the system and has minimal functionality.

Super-admins are not escrowing users; they do not involve at all in the key recovery. The super-admin can only initiate the state of escrow login. Escrow users also have escrow groups where they can share their key with the other escrow user in case an escrow user leaves the organization temporarily or permanently.

A decent candidate for an escrow group may be:

  • One of the super-admins
  • A highly trusted person in the organization
  • An individual from a different department

In this way, the chain of trust is vital, coming from three fundamentally different parts of the organization.

The Vault

Administrators can use the vault best by organizing the folder structure. It is best to avoid a vast number of sub-folders. Main folders may be divided into research areas by allocating resources for each research team to their specific folder.

note

For each created group, a folder will be automatically created in the vault with the group's name.

tip

Administrators can tag new folders in the vault with the classified project name if they want to set up a restricted folder linked to the classified project.

We recommend adding folders in the vault after setting up the teams and groups.

User Profiles vs. Permission Management

There are numerous functionalities in tiCrypt when it comes to roles. Each role has various permissions, which will change how the world looks for each user.

For the best practice, we strongly discourage users from playing with specific permissions; unless there is a particular case where they need them to carry out a complex project. We advise administrators to follow the simple User Profiles, which helps them automatically segment the user roles with their respective permissions.

User profiles are historically better for researchers due to simplicity and fast-forward setup.

They serve a similar purpose both for users and admins.

Teams

Teams are the element of administration and visibility. Users should know about each other and have clarity about their available resource levels.

Teams visibility is based on user roles as follows:

  • Sub-admin will see the users they manage in the system
  • Admins will see other admins, sub-admins, and their managed users
  • Super-admins will see all the users in the system

For best practice, we recommend discussing the management plan before giving admin access to local projects.

We also recommend having three super-admins as an appropriate management solution. Suppose a super-admin cannot access the machine or continue working for the organization. In that case, the system can use the other super-admin keys to transfer the person's key to a new super-admin securely.

Groups

When an admin creates a new group, a folder with the group's name is automatically created in the vault. The new folder in the vault serves the purpose of effectively sharing information and data with the right people. Groups facilitate and segment user access.

Minimum privilege principle applies to both groups and projects. Only explicit sharing works (E.g., PI shared a file with the whole group explicitly). Only an existing group member with active permissions can give access to a new group member.

There is significant differences between Teams and Groups as seen below:

  • Teams serve the management purpose; Groups serve the project purpose
  • Teams and the Projects are access controlled; Groups and the VMs are cryptographic
  • There can be a Team of users where each of them belongs to a different Group
  • There can be a Group where each member belongs to a different team
  • Groups can have a tag with restrictions per project; Teams cannot have a tag attached to them
  • Groups serve the success of a project; Teams serve the management of users across the organization

Administrators should keep in mind the above functionalities when creating a group. They should assign the group to a project and have the members know each other.

Projects

Projects are an organized way to enforce compliance.

In principle, users can access project files, but because of the project membership, they cannot view or download them. This operation is a vital compliance effort that may explain why a project is disabled. E.g., "the HIPAA certificate expired."

A real example is the NIH projects that have restrictions; they need a stamp on the project that states users cannot access the individual file. tiCrypt projects enforce access by mechanism. Administrators should make full use of the tiCrypt mechanism to manage project restrictions. Security levels make it easy for administrators to 'stamp' a project via user certificates.

tip

Projects can tag Groups.

Project memberships will prevent users from tagging projects; they can receive permission to tag only from the administrators. All projects may deter users from grabbing encryption keys.

Projects may be not only access-controlled but also cryptographic in combination with Groups. If a project tags a group, users can only get the key if they convince the system that they are actively part of that group and the information is relevant to them. A user must have access to the Group key to join the group, but only a group member can give the Group key to the new user.

When an admin creates a group, it also creates a current version of the group key, but each user has a private cryptographic key as part of the group key.

note

Group keys are not stored on a user's local machine. The only key stored is the user's private key.

Composable Security

Projects are entirely under admin control and backend control.

You can combine groups with projects to get the highest level of security. This is known as composable security.

Projects are not cryptographic; they are a complete mechanism. They can take away permissions from users. E.g. "the ability to add the private key to the group key".

Deleting a project is not a good practice because the resources users have accumulated on that project may be an asset in the future. It is better to park the project, a new feature coming soon into tiCrypt.

As a good practice example, several organizations use tiCrypt to tag files with HIPAA in projects whenever there is a slight assumption of becoming a HIPAA type of information.

HIPAA is a political regulation with various counterintuitive and bizarre expectations. As a result, safe-keeping projects and files under the HIPAA tag enforces the compliance mechanism; hence it diminishes the chance of ever being penalized for data-handling processes.

Project Size Scenario

Supposing you are a complex organization with three kinds of project levels, all running at the same time as follows:

Project A – Large project with 100 people involved

Project B – Medium project with 30 people involved

Project C – Small project with 6 people involved

You would have to decide which project will need User Profiles the most to control the users and the management workflows.

Project C

It is not a critical case for using User Profiles since members are limited and they work closely and well-coordinated.

Project B

This kind of project should consider using User Profiles due to the project size. Members may change with new members across a short period; updates often need reviewing; coordination is necessary for the project to be successful.

Project A

Having User Profiles set up before opening the project is critical. A large number of members requires larger management teams. The audit team is actively involved, and there is high responsibility from the project's leaders to ensure every new member fits into pre-determined User Profiles and has clear instructions to carry out their tasks successfully.

Security Levels

Security levels (e.g., HIPAA) are containers for requirements (e.g., HIPAA requests). When admins have a new project coming, they should first define the constraints and restrictions.

Constraints are the easiest way to assess a project's potential and be able to get a grant investment for it. Restrictions play a similar role in achieving project compliance. tiCrypt allows admins to decide every time if any current security levels fit the new project requirements.

In this way, system admins have leverage over future projects that require similar requests.

Security Requirements

We recommend creating security requirements after setting up security levels. Each security requirement should address one security level.

User Certifications

User Certifications should be user addressed and should have expiry dates. If a user leaves the project or the organization, admins should immediately decertify the user from the individual project or make the user inactive.

In a hypothetical situation, administrators may be in a meeting trying to help organization members achieve a healthy project goal. However, the call gets rejected for some reason. Three months later, members returned to the administrators and asked how it would be possible to apply their suggestions to the project. Admins can use user__ certifications to navigate sudden changes in management decisions.

The best practice for user certifications is to consider to which user the policy and the feature address, the practical benefit of the user resulting from this policy, and finally, the problem that this policy solves for the user.

Adding A User Checklist

After successfully adding a new user, you may follow the following checklist example for good practice:

  • User registers account
  • Sub-admin logs onto tiCrypt and updates the user's settings
  • Add the user to Teams, E.g. "DAC."
  • Update Role and Permissions role to the user
    • Set status to "escrow, active on next login."
    • Profile to "Granite User."
  • Add Security Certificate "DAC Compliance."
  • Add a user to the project "DAC" (this is the "parent" project)

If it is a Researcher:

  • Check "Cannot download."
  • Check "Cannot view."
  • Add User to "DUA project" (this is a sub-project of DAC)
  • Add User to VM (Virtual Machine)
    • Connect to VM
    • Select the "User" role

tiAudit and Reports

The following is a story from 2017 when Equifax was doing a security review on its credit history for the last four years. They noticed unfamiliar access, which turned out to be a data breach going on for the last three to six months. They, fortunately, fixed it.

Several months later, they tried to find the complete set of stolen data, but it was impossible to know what had happened. As a result, Equifax had to say by law that all databases of 144 million records had been breached. The hacker was never found.

We learn that the government is hardly convinced about any internal audits in such situations, which leaves the door open for external audit data companies, high losses, and record fines.

In tiCrypt, all actions are audited at all times into tiAudit, releasing the need to hire internal auditors or pay external data audit companies.

tiCrypt can take any file that 'smells' sensitive and non-compliant and label it as sensitive and compliant using classified project tags. We recommend that admins train users to tag files if they feel the information becomes too keen to be unprotected for compliance purposes.

tiAudit has three parts:

Alerts

Designed to help compliance officers track compliance in real-time.

note

All alerts are triggered from the system.

Reports

They are the most creative part of the auditing because they allow full customization. For good practice, we suggest having regular reports over once-a-month reports on auditing.

Users have all they need to be compliant in the reports. At a press of a button, all reports are system-generated. We recommend that the audit team use reports and queries to get deeply accustomed to tiAudit.

Queries

We strongly recommend using the queries for forensic operations. The audit team may inquire about the system database about incident responses. They can find out what happens anytime in the past since the installation of the system. Due to the depth of data, we advise this kind of investigation to be carried out by a system engineer; otherwise, it might be overwhelming to ask the proper data queries.

Universities would spend millions of dollars on consultancy companies without tiAudit forensic tools. For this reason, tiAudit records in detail everything from the primary system.

info

tiAudit is not a backup management system. It memorizes the user action, not the user content.

Although Personal Investigators may use tiAudit to spy on their users' work progress, they primarily use it for compliance at the organizational level.

Due to its past detailed queries, tiAudit can effectively predict what will happen with the data in the future, giving room for data management decisions in the present. Administrators should fabricate queries from various data angles to make the best future decisions.

Example:

For R1 and R2, universities running projects that require HIPAA stamps, tiAudit ensures the data to avoid legal penalties. By law, federal entities can release fines even for one HIPAA violation. However, the costliest part is not acceptable but determining what happened with the data. This cost may result in millions of dollars spent on third-party audit companies and data digging.

Although tiCrypt does not know what is in the data, it proficiently understands how the data moves at all times. By writing a query in tiAudit, admins can find precisely what moved with the data, saving to university millions of dollars that would have otherwise gone as legal penalties and payments to authorized auditing companies.

· 7 min read
Alin Dobra

Secure virtual machines in tiCrypt run in almost complete isolation from each-other and the environment. The main motivation for this isolation is providing a high degree of security.

Setup

In this blog we will refer to a number of tiCrypt components that participate in the activities described here. They are:

  • ticrypt-connect runs the application on user's device
  • ticrypt-frontend runs in the browser on user's device and is served by ticrypt-connect
  • ticrypt-backend is the set of tiCrypt services running on the Backend server
  • ticrypt-rest is the entry point in ticrypt-backend and uses REST protocol.
  • ticrypt-proxy mediates communication between components by replicating traffic between two other connections.
  • ticrypt-allowedlist mediates access to external licensing servers
  • ticrypt-vmc controls the secure VM and is responsible for all VM related security mechanisms

Overview

The VM security is provided through the following mechanisms:

  1. Reverse mechanism to talk to tiCrypt frontend using ticrypt-proxy component
  2. The only opened ports for a VM are port 22 for traffic tunneling from tiCrypt Connect
  3. Special mechanism to reach port 22 of VMs but no general mechanism to reach any other port from tiCrypt-backend
  4. Strictly controlled outgoing traffic through ticrypt-allowedlist component

We take each mechanism in turn end provide details on how it works.

ticrypt-proxy mediated VM communication

Due to the fact that all ports are blocked except 22 and that ticrypt-vmc (VM controller) control this port for port forwarding, there is no mechanism to directly contact a secure VM using direct access. Instead, a mechanism fully controlled by the ticrytp-vmc is employed. The mechanism relies on ticrypt-proxy (component of ticrytp-backend) to mediate communication with the user in the following manner:

  1. The ticrypt-vmc creates a web-socket connection with ticrypt-proxy at all times.
  2. When a user wants to connect to a VM, the ticrypt-frotnend creates a web-socket to match the existing VM one in ticrypt-proxy. All traffic between the two matching web-sockets is replicaed by ticrypt-proxy. Imediately ticrypt-vmc creates a new web-socket for future communication.
  3. The first message sent by both the ticrypt-frontend and ticrypt-vmc are digital signature proofs of identity and Diffie-Helman secret key negotiation.
  4. If the digital signature fails or if the user is not authorized (as determined by ticrypt-vmc) the connection is immediately closed. Similarly, if the VM validation fails, ticrypt-frontend will close the connection
  5. All subsequent messages (in both directions) are encrypted using hte negotiated key. Except the setup messages, all traffic is hidden from ticrypt-proxy and any other part of the infrastructure.
  6. All commands, terminal info, keys, etc are send through this encrypted channel.
info

The only message sent in the clear to the ticrypt-vmc contains both authentication (digital signature) and key negotiation. Any other message or functionality is immediately rejected.

info

The communication does not rely on listening on regular ports and can only be mediated by ticrypt-proxy.

info

The public key of the VM owner cannot change after learned by the ticrypt-vmc thus hijacking the communication requires a compromise of user's private key.

tip

Most of the VM functionality only requires this communication method and can only be performed using this mechanism. The only exception is application port tunneling.

Application port tunneling

In order to allow reach application deployment in VMs, a generic mechanism that tunnels TCP traffic on specific ports from the VM to user's device is provided. The mechanism is highly controlled (as explained below) but, in principle, can make any network application access possible.

note

The application tunneling is similar to the SSH tunneling mechanism used for reverse port forwarding but uses TLS

The communication pathway relies on an number of independent segments, all of which can prevent the mechanism from functioning. All traffic on these segments is proxied/forwarded and overall is secured with TLS encryption and authenticated with digital signature (not passwords). At no point any intermediate component can intercept the communication without breaking TLS.

note

Access to VMs is limited to port 22, not SSH as a service. ticrypt-connect mediates port forwarding of any desired port using the mechanisim described with ticrypt-vmc listening on port 22. Specifically, port 3389 for RDP can be forwarded. Usually, multiple such ports are forwarded to allow richer functionality.

Communication setup

Initiating the forwarding tunnel setup requires the following steps:

  1. The ticrytp-frontend using authenticated session, tells ticrypt-rest (entry point of ticrypt-backend) to create the pathway to a specific VM
  2. The request is validated and ticrypt-proxy is informed of the operation
  3. The ticrypt-proxy sets up a listening endpoint on one of the designated ports (usually 6000-6100). The endpoint is strictly set up and only allows access from the IP address used to make the request.
  4. The ticrypt-frontend is informed of the allocated port.
  5. ticrypt-frontend asks ticrypt-connect to generate a TLS certificate for authentication and connection encryption.
  6. ticrypt-frontend tells ticrypt-vmc to accept the application forwarding and provides the authentication TLS certificate
  7. ticrypt-vmc replies with a list of ports that need to be tunneled
  8. ticrypt-frontend tells ticrypt-connect to start the connection and use the previous certificate.
  9. ticrypt-vmc, upon connection creation, checks that the digital signature is correct and initiates the TLS mediated tunneling.
  10. The traffic to/from local port on user device is tunneled and re-created within the VM to allow application access
tip

A special case is provided for SFTP over port 2022 if the feature is enabled in ticrypt-vmc. It can be used to move large amount of data from user's device to the VM.

Communication pathway

The communication pathway for port tunneling is:

  1. The ticrypt-connect accesses the allocated port controlled by ticrypt-proxy. The connection is pinned to the IP address the setup request comes from.
  2. ticrypt-proxy forwards the request to the VM Host endpoint (explained below)
  3. The VM Host endpoint forwards the traffic to port 22 of the correct VM
  4. ticrypt-vmc listens on port 22 and runs the TLS protocol with port tunneling.
tip

Port 22 is specifically selected to ensure no SSH server is deployed instead to allow normal access to the VM. The traffic on 22 is the TLS protocol with ticrypt-vmc control and not SSH protocol.

Secure VM isolation

In order to ensure VM security, only port 22 is opened inbound. Furthermore, all outbound traffic is limited to ticrytp-rest traffic unless an exception is provided by ticrypt-allowedlist (next section)

The specific mechanisms deployed for VM isolation are:

  1. All traffic to ports other than 22 is blocked internally by the VM thus isolating any other access. Note that application access is provided by the port tunneling above.
  2. All traffic to the VM ip on ports other than 22 is blocked using firewall rules on the VM host.
  3. External traffic to VMs is not routed by the VM host (with the exception of port 22)
  4. Traffic from VMs to outside is blocked unless ticrypt-allowedlist mechanism is used
info

The specific mechanism to allow access on port 22 but no other traffic is to block any traffic routing but provide a forwarding mechanism from a range on the VM host to port 22 of the IP ranges dedicated to the secure VMs. There is simply no way from a server outside the specific VM host to access any other VM port.

tip

The outbound traffic from secure VMs is blocked since it can be used to exfiltrate data thus resulting in data breaches.

Licensing server access

In order to use most commercial software, access to external licensing servers needs to be provided. Most of the time, the licensing servers reside within the organization, but occasionally, the servers are external. Recognizing this, tiCrypt provides a strictly controlled mechanism to allow access to licensing servers.

The ticrypt-allowlist component mediates setting up the mechanism that allows access to licensing servers. It does so by manipulating:

  1. Firewall and port forwarding rules on the server running ticrypt-backend
  2. DNS replies to requests that VMs make

The mechanism is strict and very specific. Unless a mapping is provided using ipsets and firewall rules, any outgoing traffic from VMs is blocked.

The ticrypt-allowlist component works in conjunction with ticrypt-frontend to provide a convenient way to enable/disable access. This mechanism requires SuperAdmin access.

· 3 min read
tiCrypt Team

Administrator roles

There are three different administrator roles within tiCrypt; super-admin, admin, and sub-admin. Just like everything else in tiCrypt, there is a need for security which is implemented with an administrator hierarchy.

Super-administrator

A super administrator is the highest role in the system. This user can do ANYTHING in the system, however, the main purpose of a super-admin user is to manage global or system-wide settings such as deployment settings. Superadmins are also responsible for accessing and restarting system services. Super-admins can modify all types of users. This role is limited to a very select few administrators within the organization. Again, all admins (including super admins) do not have access to the user data unless a user explicitly shares the data with the admin.

Administrator

Admins are managers of the entire system. This role is defined as a traditional system manager role with no access to user data unless explicitly shared. Admins can only modify basic user and sub-admin permissions. Admins cannot modify super-admin permissions nor can they do the tasks specific to super-admins stated above.

Sub-administrator

Sub-admins within the system are the lowest level of administrators. A sub-admin role allows specific users to act as administrators within the tiCrypt system for their team. Admins in the system assign a managed object to the specific sub-admin.

The sub-admin has access to the Management tab within the tiCrypt system, just like the admins above, but sub-admins would only list members, VMs, projects, etc., associated with the admin’s assigned team. This allows for a sub-admin to activate team members, change team members' permissions, manage tiCrypt projects, or any other admin action for that specific team only.

Three guiding principles act as rules for a sub-admin.

If a user is deactivated and belongs to no team, a sub-admin can place this new user into their team. This allows for sub-admins to onboard and activates users without the need for a Super Admin/UMD IT Admin. This rule prevents the sub-admin from managing already existing members in the system that are not part of the defined team.

If a user explicitly belongs to a team, then the sub-admin of that team can manage that user. If a user is removed from a team and is no longer a member of any team, that account becomes deactivated, and default permissions are restored. Once the account is deactivated, a Super Admin/UMD IT Admin will need to change the role. This rule is in place to prevent possible malicious permission changes.

Sub-admins can create new teams, but new teams will have a default quota. The quota will need to be increased by the Super Admin/UMD IT Admin. This will prevent sub-admins from over-utilizing (or over-allocating resouUMD ITes) in the system without permission from UMD IT.

· 5 min read
Alin Dobra

What is hyper-threading

Hyper-threading is a set of technologies introduced by Intel in 2002 that allow "double-personality" for cores. DEC initially developed and inspired much of the technology for Alpha processors. AMD provides similar technology but calls it "simultaneous multithreading" since Hyper-threading is an Intel trademark.

note

A good introduction to Intel hyper-threading: Intel's Hyper-Threading by Tinker and Valerino

Primer on the core architecture

The way modern all processors, starting with the Intel Pentium, work is:

  • They have a set of registers: internal processor storage that is very fast
  • They have various caches (L1, L2, L3)
  • They have sophisticated mechanisms to re-order and parallelize instructions
  • They have extensive instruction sets that allow various types of processing (mathematical and logical operations, etc). More importantly, arguments to the instructions are primarily registers but can be memory references as well
  • They have execution units (up to 3 per core) that can execute instructions. The process of scheduling instructions on various execution units is complex

Execution in modern processors (without hyper-threading) consists of running instructions to change the state of registers and of memory. Access to registers is high-speed (no waiting), but memory access can be very slow (depending on the efficiency of the cache hierarchy). When a slow memory access operation is performed, the core will essentially do nothing until the memory is accessed; this is somewhat mitigated by the ability of cores to run up to 3 instructions at a time, but for the most part, it is accurate.

Key ideas in hyper-threading

  1. Create two sets of independent registers in every core; only one set is active at a time.
  2. When the execution of one piece of code on the current register set gets stalled (due to expensive memory access), switch to the other set of registers and run a different program
  3. Switching registers sets switches the context (the OS is usually responsible for this) and must invalidate the cache (exactly what the OS has to do to switch tasks)
note

All the machinery to execute instructions is shared between the 2 hyper-threads except the set of registers. As such, the two hyper-threads cannot execute at the same time.

note

The OS can switch threads on cores without the hyper-threading mechanism. The hardware would do the switch faster.

tip

The benefits of the hyper-threading are dependent on the application since it can only help if cache performance is poor

Misconceptions about hyper-threading

The biggest misconception about hyper-threading is the fact that it "doubles" the number of cores. If hyper-threading is enabled (usually in BIOS), indeed the number of cores reported to the OS is double the initial core count. The trouble is the fact that the "hyper-threaded" cores do not have the performance of the original cores. The section above explained that only the registers are thread-specific; everything else is shared. As such the performance of two hyper-threaded cores depends fundamentally on getting stuck on memory accesses.

The second misconception is related to the fact that it cannot hurt, even if it does not provide full benefit. For certain workloads (see the section below), hyper-threading can add a 5-10% performance penalty. It also makes thread scheduling more complicated (the two hyper-threaded cores that are "related" should not run programs at the same time unless strictly needed).

caution

Hyper-threaded technologies have been hyped up by both Intel and AMD over the last 2 decades. There are a lot of marketing "stories" around the technologies that do not necessarily translate to actual performance. Processors capable of hyper-threading are targeting enterprises (and consumers more and more); they are usually more expensive than processors without the capability.

Real hyper-threading performance

The only way to determine if hyper-threading is beneficial in specific circumstances is to turn it on/off and benchmark the applications. The interplay between thread switching, mapping to underlying execution units, and cache performance is too complex.

Some of the relevant studies/benchmarks on hyper-threading and their main findings are:

Pitfalls of using hyper-threading for VM Hosts

  1. Will not result in significantly better performance overall (11% at best)
  2. Will double-book the actual processors and offer half-performance to the users. Most VM users will perceive this as a performance degradation since most of their tasks will run at half-speed. This is very problematic since users are very per/core performance sensitive.
  3. Has added security risks Spectre-like attack
  4. The CPU can be over-subscribed for VM without hyper-threading since the VMs run as processes in the underlying OS.
tip

Overall, enabling hyper-threading on VM hosts adds too few benefits and comes with too many pitfalls to recommend for production work.

· 7 min read
Alin Dobra

tiCrypt Vault Storage

The tiCrypt Vault offers a file system-like facility that allows files and directories to be created and used. All the metadata, such as file properties, directory entries, access information, decryption keys, are stored in the MongoDB database used by the tiCrypt-file storage service.

In tiCrypt Vault, the file content is broken into 8MB chunks, each encrypted independently using the file key. The size of such chunks is 8MB+64Bytes on disk (unless they are the last, incomplete chunk ). The extra 64Bytes contain the IV (initialization vector for AES encryption). For each file, the chunks are numbered from 0 onwards. Based on the file ID, the chunks are stored in a directory structure, visible only to the tiCrypt backend (preferably not to the VM hosts). The location of the storage can be configured in the configuration file of the tiCrypt-storage service.

tiCrypt is not opinionated on the file system or integration with other systems for this storage but, for compliance reasons, it is recommended that the access is restricted to only the tiCrypt backend.

Without the decryption keys that only users can recover, the content of the chunk files is completely indecipherable. It is safe to back up these files using any method (including non-encrypted backup, cloud, etc). The strong encryption coupled with the unavailability of the key to even administrators ensures that this content can be replicated outside the secure environment from a compliance point of view.unavailability of the key to even administrators, ensures that from a compliance point of view, this content can be replicated outside the secure environment.

tiCrypt Encrypted Drives in Libvirt

tiCrypt virtual machines use a boot disk image that is not encrypted and one or more encrypted drives. Both disk images and encrypted drives are stored as files in the underlying distributed file system available to all Libvirt host servers. The specific mechanism uses the notion of Libvirt disk pools. Independent disk pools can be defined for disk images, encrypted drives, drive snapshots (for backup), ISOs, etc. Each pool is located in a different directory within the distributed file system.

Libvirt (and tiCrypt, by extension) is agnostic to the choice of the file system where various disk pools are defined. A good practice is to place the various disk pools on a distributed file system visible (preferably in the same location) and to mount the file system on all VM hosts. Any file system can be used, including NFS, BeeGFS, Luster, etc.

As part of the virtualization mechanism, Libvirt makes the files corresponding to drives, stored on the host file system, appear as devices to the OS running within the VM. Any writes to the virtual device get translated into changes of the underlying file. The situation is somewhat more complex when snapshots are used since multiple files on the disk will form the virtual device.

Encrypted Drive Creation

Upon drive creation, tiCrypt instructs Libvirt to create a drive. This results in a file being created in the underlying file system in the corresponding drive pool. Two types of drives are supported: raw, with extension .raw, and QCOW2, with extension .qcow2.

For a raw drive, a file as large as the indicated drive size gets created. The file content is initialized with 0s (corresponding to a blank drive). Writes in the virtual drive result in writes in the corresponding file at exactly the same position (e.g., if block 10244 of the virtual drive is written, block 10244 of the raw file gets changed as well)

For a QCOW2 drive, only changed blocks get written; the file format is quite complex and supports advanced features like copy-on-write. The initial file size is small (low megabytes) when the drive is new; the file grows in size as more data is written to the disk.

The qemu-img tool can be used to convert between two formats. Usually, tiCrypt takes care of setting the drives up without the need for this tool.

A newly created tiCrypt disk is blank. Absolutely no formatting of the drive or any other preparation has been performed. The drive will be formated the first time it is attached to a tiCrypt virtual machine. The main reasons for this are:

  • The encryption/decryption key for the encrypted drive is kept secret from the infrastructure. This includes the tiCrypt backend, Libvirt, and underlying VM host.
  • The choice of the file system to use is delegated to the underlying operating system and tiCrypt VM Controller. Libvirt is not aware, nor does it needs to know, of the actual file system on the drive. For Linux formmated drives, by inspecting the files backing up the drives, there is no way to tell even if the drive is formatted at all, let alone any information on content or type of file system.

Encrypted Drive Formatting

As far as Libvirt is concerned, only low-level disk reads and writes exist. Whatever operation the operating system is performing gets translated into reading/write operations on the virtual disk; in turn, this will result in reading/write operations for the underlying file in the disk pool.

In Windows, a normal NTFS file system is created but immediately (before the drive is made available to the user), BitLocker is turned on. This ensures that all files created subsequently are encrypted. BitLocker uses so-called "full volume encryption," i.e., all the new data will be encrypted, including the metadata. An external tool scanning the backing file can determine that the drive is an NTFS formatted drive and read all non-encrypted content. Sine tiCrypt turns on encryption immediately, very limited information is visible.

In Linux, the LUKS full disk encryption mechanism is used. It essentially places an encryption block layer between the raw drive (virtual drive in this case) and the file system (usually EXT4). This way, absolutely all information on the disk is encrypted. An external tool can only tell which disk blocks have been written to (are non-zero) but can derive no information on the rest of the content

tiCrypt Non-secure Drives

Two types of non-secure drives are supported in tiCrypt: ISOs and read-only NFS shares.

Attaching ISOs

ISOs are made available using read-only CD-ROM devices. As such, they are always safe to mount in a secure tiCrypt VM. Both Linux and Windows can readily mount such ISOs and make them available as "drives."

ISOs are particularly useful if the NFS shares described below are not used. For example, Python or R packages could be made available as ISOs so that various VMs can install the required packages locally.

Attaching NFS file systems

By allowing, through firewall rules, access to a local NFS server, various tiCrypt VMs can mount a common file system for the purpose of accessing public (non-secure) data, packages, software, etc.

From a security point of view, the NFS server should export the data as read-only. The tiCrypt secure VMs should never be allowed to mount a read-write NFS share since data can be exfiltrated, thus defeating the great effort put into tiCrypt protections against data exfiltration. This will most surely make the tiCrypt deployment non-compliant.

A further restriction is related to the location of the NFS server. The NFS file system must be under full control of the system administrators of tiCrypt. It has to be part of the tiCrypt system envelope; for example, one of the server's part of the tiCrypt infrastructure can take this role. This restriction is due to compliance complications: the security envelope extends to all parts of the system; a remote NFS server becomes part of the secure environment and is subject to all the security restrictions.

A practical recommendation is to create a local NFS server inside the security envelope and regularly sync content from an external NFS server. System administrators responsible for the operations need to be identified in the security plan (compliance requirement).

· 5 min read
tiCrypt Team

Business case

This blog post discusses what a configuration file is and why you, as an admin, should care about it when running your virtual machines. A configuration file can alter the internal settings of your virtual machine.

Why do we need this?

When the administrator creates a VM Image, they need two things: the tiCrypt software to manage the image, and the configuration file. tiCrypt provides users with a default configuration file. The administrator can use the tiCrypt config file, or use their own. Allowing users to upload their own configuration files allows for customization for what the researcher needs.

What types of parameters are in a config file?

Below is the default configuration file with notes on what each object and its parameters mean.

[terminal]

  • enabled = false
    Whether or not the terminal service is enabled. On windows, the terminal is powershell and on linux it will be linux.

  • #command = "/bin/bash"
    The default command to use when running terminals in linux.

  • #command = "powershell.exe"
    The default command to use when running terminals in windows.

  • scrollback = 10000
    Default number of lines of terminal scrollback history kept.

[tunnel]

  • enabled = false
    Whether or not the tunnel service is enabled.

  • serverPort = 22
    TCP port on which to bind the tunneling service.

  • allowedPorts = [] This is our default port. We use a list of ports for tunneling. The following are examples of what users can set allowedPorts to.

  • allowedPorts = 5901
    A single port can be used.

  • allowedPorts = "5901-5905"
    A range of ports can be used.

  • allowedPorts = [ 14500, 5901 ] An array of ports can be used.

  • addGroups = []
    This is our default list. addGroups is a list of additional system groups that users with tunneling permissions will be added to.

  • addGroups = [ "Remote Desktop Users" ]
    Used by Windows ONLY: allow access to RDP

  • idleTimeout = "15m"
    This is the timeout for idle tunnels. It defaults to 15 minutes. If set to positive duration, tunnels without active forwarded connections will be killed after the specified timeout. The minimum non-zero idle timeout is 1 second.

  • sftpEnabled = false
    Whether SFTP support is enabled. If enabled, an SSH daemon will be run that is configured to only allow SFTP connections. Enabling SFTP allows for one way SFTP from the local client to the virtual machine.

  • sftpPort = 2022
    The local port on which the SFTP SSH daemon runs. This will be automatically added to the allowed tunnel ports.

  • sshDirPath = ""
    An sshPath is NEEDED if SFTP is enabled. The path to the directory containing the sshd(.exe) and ssh-keygen(.exe) executables. If not set, the following will be checked for the executable:

    1. The assets archive at bin/ssh/
    2. The system path

[tunnel.services]

  • xpra = 14500

Optional names for ports, which may be referred to in the connection instructions for the VM. This is only needed for linux. This will be ignored windows. It is recommended to use the xpra information that we provide. More information can be found here

[tunnel.cert]

  • country = "US"
    If specified, country MUST be a two-letter country code.
  • organization = "unspecified"

[users]

  • changeAdminPassword = false
    If true, the VMC will attempt to change the admin password at startup to a random password. This prevents anyone from knowing the password.

  • managersAsAdmin = false
    If this is set to true, then IN WINDOWS, managers or owners of the VM can fulfill admin tasks without a password. If this is true in LINUX, then managers will be part of a group called "sudoers" and can act as admins without a password.

  • createDirs = []

  • createHiddenDirs = []
    These two parameters apply to Windows ONLY. Users can add the names of directories to be automatically created in the user's home on their encrypted drive if they do not already exist. This will allow for the directories to be automatically linked into the user's profile on the C: drive even if they did not originally exist.
    If it is left blank, everything in the home drive will be a junction. Both only pertain to Windows.

[commands]

  • commands.rootCommands
    Commands that are run by the root upon startup

  • commands.rootCommands.runOnlyOnceCommands
    Commands that are run only once

  • commands.rootCommands.runEveryTimeCommands
    Commands that are run every time event_name = {command0_name = "command0", command1_name = "command1"}

  • commands.userCommands
    Commands to be run by the user

  • commands.userCommands.runOnlyOnceCommands
    Commands that are run only once

  • commands.userCommands.runEveryTimeCommands
    Commands that are run every time

tiCrypt Default Config

[terminal]
enabled = false
command = linux = "/bin/bash" windows = "powershell.exe"
scrollback = 10000
[tunnel]
enabled = false
serverPort = 22
allowedPorts = []
addGroups = []
addGroups = [ "Remote Desktop Users" ]
idleTimeout = "15m"
sftpEnabled = false
sftpPort = 2022
sshDirPath = ""
[tunnel.services]
xpra = 14500
[tunnel.cert]
country = "US"
organization = "unspecified"
[users]
changeAdminPassword = false
managersAsAdmin = false
createDirs = []
createHiddenDirs = []
[commands]
[commands.rootCommands]
[commands.rootCommands.runOnlyOnceCommands]
[commands.rootCommands.runEveryTimeCommands]
[commands.userCommands]
[commands.userCommands.runOnlyOnceCommands]
[commands.userCommands.runEveryTimeCommands]

· 8 min read
Alin Dobra

Core principles

The main idea is to introduce two notions:

  • full backup/checkpoint is a complete backup of the resources. Recovering the resource only requires the information in such a backup.
  • incremental backup is a "delta" change from previous incremental backups or checkpoints. All the incremental backups between the recovery point and a checkpoint are required to recover the resources covered by the backup.
  • backup strategies specify what type of resource to backup and how often.
  • backup domains specify specific resources to back up together with the backup strategy. All the resources covered by a backup domain are backed up together.
  • backup: an incremental or full backup of a specific backup domain at a specific point in time.
  • external backup server: server with SFTP connectivity that holds the backup files externally.
  • backup solution: Software running on the external backup server that actually backs up the files on tapes/cloud.

Except for SFTP interface availability, there is no requirement on the external backup server. In particular, any operating system and backup solution can be used.

Backup strategies

  • name: displayable name for the backup strategy
  • id: internal ID
  • incrementalInterval: the interval between incremental backups in seconds.
  • checkpointInterval: the interval between full backups/checkpoints in seconds.
  • teamStrategy: the strategy for the teams
  • projectStrategy: the strategy for the projects
  • userStrategy: the strategy for the users

The strategies are maps from names to boolean (true/false). The possible names and meanings are:

  • userVault [team,project]: should we backup the vault of the users part of the team/project?
  • userForms [team,project]: should we backup the forms of the users in the team/project?
  • groups [user, team, project]: should we backup the Vault related to groups of the user/team/project?
  • drives [user,team,project]: should be backup the drives (marked as backup) of the user/team/project?
  • vault [user]: backup user vault
  • forms [user,project]: backup user/project forms

Backup domains

Backup domains are simply collections of users/projects/teams that need to be backed up together.

The backup domains can be set up at sub-admin+ level and do not require full admin access. Only resources that the sub-admin has access to can be included in a backup domain. Specifically:

  • if a sub-admin manages a team, the team can be added
  • if a sub-admin manages a project, the project can be added
  • if a user is in a team or a project managed by a sub-admin, the user can be added

At a high level, the backup domains specify:

  • name: displayable name of the domain
  • owner: Who owns the domain
  • managers: Other users that can manage this domain
  • sftp: SFTPSpec object
  • strategy: ID of the backup strategy to use

The SFTPSpec object has the following structure:

interface SFTPSpec {
server: string, // the name or IP of the sftp server
port: number, // the port, default 22
user: string, // sftp user name
directory: string, // the directory where backups are stored
}

The objects in the domain to backup are specified using a "membership" model. Specifically, a CRUD interface that allows listing, adding, and deleting objects to a domain is needed. Listing and changing the membership can be done by specifying:

interface BackupDomainSpec {
domainID: string, // the ID of the domain
type: "user" | "team" | "project", // type of the object
objID: string, // Object ID
}

Backups

Backups can be used for two different reasons:

  • disaster recovery: In situations when the system needs to be re-created from scratch (disastrous failure of storage, accidental system deletion, system cloning)
  • time machine: Recovering the state of a resource from the past, e.g. state of a drive as of 3 days ago, recover deleted file from 2 weeks ago, etc.

Some important principles are:

  • security: the backup is encrypted to the full extent possible. Neither the external backup server or backup solution used should be trusted.
  • minimize traffic: only the files strictly needed to perform recovery from backup should be required. This ensures minimal costs for recovery

Creating a backup

A backup consists of:

  1. Entries in the ticrypt-backup service database indicating information on what is backed up and where
  2. Files placed on the tiCrypt backend server, later transferred to the remote sftp backup server indicated by the backup domain metadata.

Creating a backup will involve the following steps:

  1. A backup directory is created to host the metadata and files for the specific backup of the specific backup domain
  2. Server computes which resources (files, forms, drives) are backed up. For "full" backups, all such objects permitted by the backup strategies are listed, for "incremental" backups, only objects that changed in the backup period are listed
  3. Server prepares a "metadata-file" containing all the relevant backup information and the list of backed up files with the corresponding auxiliary files
  4. For each type of object, auxiliary files are copied into the "backup directory"
  • files: The file chunks
  • forms: the form entries into an SQLite database
  • drives: the .qcow drive image (full backup), or the image snapshot (incremental backup)
  1. Metadata to be able to recover the information about the resources is added to the metadata file. Specifically:
  • files: file metadata and all file keys
  • forms: form metadata and all form keys
  • drives: drive metadata and all drive keys
  1. The backup information on this specific backup is stored into ticrypt-backup service's database

Recovering a resource from the backup

This is primarily performed to recover a previous version of a resource, i.e. time machine functionality.

To recover a resource from the backup, the following steps are taken:

  1. Recover the metadata file from the backend server or remote SFTP server
  2. Determine other backup metadata files (incremental backups need access to all the metadata files of all incremental backups they depend on)
  3. Compute all other auxiliary files needed
  4. Recover the content of the auxiliary files on the system backend (via SFTP if needed)
  5. Recover the state of the resource:
  • files: recover the contend of missing chunks and the metadata
  • forms: recover the content of the SQLite database and metadata
  • drives: recover the state of the drive, by "merging" snapshots on top of the most recent checkpoint.

Recovering the entire state from the backup

This is primarily performed as a form of disaster recovery. Specifically, to recover all resources covered by the backup.

The process is the same as above but it spans all resources covered by a backup.

Caching backup files for better performance

In order to speed up recovery, the backup files can be cached at multiple levels:

  • backend server: can store, without guarantee of availability, any of the backup files/directories.
  • external backup server: same as above thus removing the need to recover from the actual backup solution.

It is more valuable to cache metadata rather than auxiliary files. The metadata is required even to figure out what can be recovered.

User stories

  • As a sub-admin+, I want to list existing backup strategies
  • As an admin+, I want to be able to add/delete/modify backup strategies. This requires admin+ level since it is based on global system policy/requirements. It is also intended to limit the choices for backups to remove decision overload.
  • AS a sub-admin+, I want to list backup domains. Depending on role and visibility, only some of the domains are visible here.
  • As a sub-admin+, I want to be able to add/delete/modify backup domains
  • As a sub-admin+, I want to add/delete other sub-admins to the list of managers for specific backup domains
  • As a sub-admin+, I want to list objects covered by backup domains. Depending on role and visibility, only some of the objects are visible here.
  • As a sub-amin+, I want to add/delete objects to a backup domain
  • As a sub-admin+, I want to list backups associated with a backup domain
  • As a sub-admin+, I want to check the status of an ongoing backup
  • As a sub-admin+, I want to initiate a backup (incremental or full) immediately on a specific domain. This might be needed to allow backups before dangerous operations. It probably requires special permission.
  • As a sub-admin+, I want to list resources that can be recovered by a specific backup
  • As a sub-admin+, I want to list backup files needed to recover a resource
  • As a sub-admin+, I want to list backup files needed to recover the full state of a backup domain
  • As a sub-admin+, I want to recover a specific resource, possibly in a "copy" of the original resource. This requires all files listed as backup files to be available on either the tiCrypt backend or the remote SFTP server.
  • As a sub-admin+, I want to recover a complete domain. This requires all files listed as backup files to be available on either the tiCrypt backend or the remote SFTP server.