Single-Tenant Titus: VM Isolation on Containers

Single-tenant Titus: VM-level isolation on a multi-tenant container platform

Editorial position: This article examines Netflix’s approach to achieving VM-level isolation within a multi-tenant container environment using its Titus platform. The focus is on the technical constraints, engineering decisions, and practical lessons that can be applied beyond Netflix’s specific context.

The constraint behind the work

Netflix operates a container platform called Titus that supports both shared and dedicated workloads. For certain single-tenant applications—particularly those with strict security or compliance requirements—the level of isolation provided by standard container runtimes was insufficient. Containers share the same host kernel, which can pose risks around resource contention, signal propagation, and kernel-level vulnerabilities.

The core constraint was to provide VM-level isolation—where each workload runs in its own kernel space—without sacrificing the flexibility and efficiency of a container-based system. This meant finding a way to combine the benefits of containers (rapid startup, lightweight footprint) with the strong isolation properties of virtual machines.

The engineering choice

To address this constraint, Netflix developed single-tenant Titus, a specialized execution environment that runs each single-tenant workload inside a lightweight virtual machine. These VMs are provisioned on-demand from Amazon EC2 instances, with each instance hosting a small number of VMs to optimize resource utilization.

The key engineering decisions include:

• VM provisioning: Using Kubernetes-style controllers to manage VM lifecycle from boot to termination.
• Kernel customization: Building minimal, hardened kernel images tailored to specific workload requirements.
• Networking model: Implementing VPC-level networking within each VM to mirror traditional VM isolation.
• Integration with existing workflows: Ensuring that deployment pipelines, monitoring, and logging remain consistent with multi-tenant Titus workloads.

How it works

Single-tenant Titus operates by launching a small number of VMs per host EC2 instance. Each VM is allocated a fixed set of CPU and memory resources, and runs a minimal container runtime (such as Docker or containerd) to execute the application containers.

The platform uses existing tools and patterns, including:

• Kubernetes operators to manage VM provisioning and lifecycle.
• Custom resource definitions (CRDs) to represent VMs and their state within the cluster.
• Cloud-init or similar tools to configure VMs at boot with necessary security policies and application artifacts.

From the developer’s perspective, deploying a single-tenant workload involves specifying isolation requirements in the pod spec, after which Titus handles the VM allocation and container launch.

The scale assumptions

This design assumes a moderate scale environment where the number of single-tenant workloads is relatively small compared to shared workloads. Each VM consumes additional resources (memory for the kernel, overhead for hypervisor), so packing efficiency decreases as VM density increases.

Netflix’s implementation is optimized for bursty, elastic workloads typical in a streaming service, where instances can be rapidly scaled up or down based on demand. The system assumes that EC2 capacity is available to provision new VMs quickly.

Operational lessons

Operating a hybrid container-VM environment introduces several operational considerations:

• Increased complexity: Monitoring now spans both container metrics and VM health, requiring unified tooling.
• Resource accounting: Precise tracking of VM and container resource usage is necessary to avoid overcommitment.
• Security patching: VM images must be kept up-to-date with kernel patches independent of host OS updates.

In practice: Teams should invest in automation for VM lifecycle management and integrate security scanning into CI/CD pipelines.

Trade-offs the source accepts

Netflix accepts several trade-offs to achieve VM-level isolation:

• Higher resource overhead: Each VM consumes additional memory and CPU compared to a bare container.
• Longer startup times: Booting a VM takes longer than starting a container, affecting latency-sensitive workloads.
• Reduced density: Fewer workloads can run per host when using VMs, increasing infrastructure costs.

These trade-offs are deemed acceptable for workloads where isolation is a higher priority than density or speed.

What smaller teams can reuse

Smaller teams facing similar isolation requirements can adopt parts of this approach without full replication:

• Use lightweight VMs: Tools like Firecracker or gVisor provide VM-level isolation with smaller overhead.
• Adopt Kubernetes operators: Implement custom resources to manage VM lifecycle within existing clusters.
• Maintain consistent tooling: Keep deployment, monitoring, and logging pipelines aligned across container and VM workloads.

Watch out: Smaller teams should evaluate whether the isolation benefits justify the added operational complexity and cost.

Who should act

This approach is most relevant for organizations that:

• Have workloads with strict compliance or security requirements.
• Operate at a scale where occasional VM provisioning does not impact overall efficiency.
• Possess the operational maturity to manage a hybrid container-VM environment.

Bottom line

Netflix’s single-tenant Titus demonstrates that VM-level isolation can be integrated into a container platform to meet specific security and compliance needs. While it introduces trade-offs in resource efficiency and complexity, the pattern offers a practical way to balance isolation and flexibility. Teams facing similar requirements can adapt elements of this design to their own infrastructure.

Sources

Single-tenant Titus: VM-level isolation on a multi-tenant container platform