Virtualization: execution boundaries and honest guarantees

“Virtual machine”, “microVM” and “sandboxed container” are not sizes on one linear scale. They place a different kernel or syscall boundary between an untrusted workload and the host, expose different device/ABI surfaces, and make different lifecycle promises.

KVM and a userspace VMM

Linux KVM exposes kernel interfaces for VM and vCPU execution. A userspace VMM such as QEMU supplies machine construction, device emulation, firmware integration, migration and policy. Saying “KVM runs the whole VM” hides the userspace device and lifecycle surface; saying “QEMU emulates every instruction” hides hardware-assisted guest execution.

Compatibility includes CPU features, machine type, firmware, devices, storage semantics and guest expectations. Pin or negotiate those inputs rather than assuming any two hosts can receive the same running VM.

Firecracker microVMs

Firecracker is a KVM-based VMM optimized for serverless and container workloads. It intentionally presents a small device model and uses a jailer plus host controls to reduce attack surface. Published figures such as VMM startup time or memory overhead are measurements under stated conditions, not fixed guarantees for an arbitrary function invocation.

End-to-end cold latency may include placement, image/kernel availability, microVM creation, guest boot, runtime initialization, application initialization and dependency warm-up. AWS Lambda may freeze and reuse an execution environment, but applications must not depend on a fixed retention duration or on every invocation getting a fresh environment.

gVisor is not a microVM

gVisor runsc implements the OCI runtime surface. Its Sentry is a userspace application kernel that implements/intercepts workload system calls, with Gofer and networking components as needed. The KVM platform can use virtualization extensions for address-space switching, but the architecture retains a process model and does not become a hardware-device VM. Syscall and device compatibility must be tested against the workload.

Kata Containers uses another design: an OCI/container integration surface backed by lightweight VMs and guest kernels. “Works with OCI” does not mean every low-level kernel feature, device, performance profile or nested-runtime behavior is identical across runc, gVisor and Kata.

Migration is a protocol, not teleportation

Precopy sends RAM while the source runs, then repeats pages dirtied during copying. If dirtying rate approaches available migration bandwidth, it may not converge within the downtime objective. Device state, storage, CPU compatibility and application health are part of the move.

Postcopy starts destination execution before every page arrives. It can handle non-converging precopy differently, but faults on missing pages depend on the source and migration channel. Failure while state is split is materially riskier. QEMU provides recovery mechanisms for some interruptions; that is not the same as loss being impossible.

Migration streams and saved device state are inputs to validate, even when normally produced by trusted infrastructure. Keep source cleanup behind destination health proof, retain a cancellation/rollback policy where the chosen mode allows it, and rehearse failure during each phase.

Isolation and performance

Every boundary can have vulnerabilities, side channels and resource contention. Hardware virtualization narrows some shared-kernel exposure; it does not remove host, VMM, firmware, device or microarchitectural risk. Resource limits and rate limiters contain noisy neighbors only to the degree actually configured and monitored.

There is no universal “VM overhead” percentage. Measure CPU-bound, syscall-heavy, I/O-heavy and network workloads separately, including p99 latency, steal time, exits, faults and host contention.

Scenarios

A general VM is created across control plane, userspace VMM, KVM, hardware, guest kernel and workload readiness.

A Firecracker start separates VMM creation from guest and application initialization.

A gVisor workload crosses the Sentry userspace-kernel boundary instead of a fictitious microVM device layer.

Precopy exposes dirty-rate convergence, compatibility and stop-and-copy validation.

A channel failure while pages are split demonstrates the explicit postcopy recovery risk.

Primary sources

• KVM API: https://docs.kernel.org/virt/kvm/api.html
• QEMU migration documentation: https://www.qemu.org/docs/master/devel/migration/index.html
• QEMU postcopy documentation: https://www.qemu.org/docs/master/devel/migration/postcopy.html
• Agache et al., Firecracker: https://www.usenix.org/system/files/nsdi20-paper-agache.pdf
• AWS Lambda execution-environment lifecycle: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html
• gVisor architecture: https://gvisor.dev/docs/