05 / Lifecycle & portability

Stop time. Save the machine. Continue elsewhere.

A pause keeps the VM in host memory. A full checkpoint persists enough CPU, memory, device and disk state to recreate that machine on a compatible Linux/amd64 KVM host.

A consistent cut across the VM

Copying RAM while a disk request is still changing buffers is not a consistent checkpoint. Saving first pauses the vCPUs, then takes the MMIO bus's exclusive I/O gate, waiting for in-flight handlers and blocking new device access while state is captured.

Full checkpoint and restore lifecycle Pause CPUs and quiesce devices on host A, collect memory and device state plus complete disks into two artifacts, transfer both, validate and restore private working disks on host B. Host A / VMpause every vCPU Quiesced stateRAM · CPU · devices Two artifactsstate.snap (compressed)+ sibling disk assets/ Host B / validateCPU + format + hashesprivate disk copies Resume executionsame guest state Full copy.Not live migration.
  1. Make vCPUs exit KVM_RUN and wait at the pause barrier. Acquire the device I/O gate before capturing memory or copying disks.
  2. Capture CPU, RAM, interrupt/timer, UART, virtio register and packed-ring state. Materialize every logical disk, then publish the compressed state file with its disk-asset references.
  3. Transfer the state file and its sibling storage directory with their relative layout unchanged. Check the format, host CPU/KVM compatibility, disk sizes and SHA-256 hashes.
  4. Restore RAM and hardware/device state, recreate network backends, and run against private copies of the saved disks. Resume the guest without booting its kernel again.
Saving preserves the source VM's previous running/paused state. Moving the saved files does not automatically stop the source.

Pause is not save; resume is not restore

  • Pause / resume: stop and restart guest CPU execution in the existing VM. RAM, devices and host resources remain allocated. A pause alone does not create a durable recovery point or freeze external services.
  • Save: persist a version 3 full-machine checkpoint, temporarily pausing execution and quiescing device I/O.
  • Start from snapshot: create a new VM using saved configuration and machine state, without the original kernel, initrd, base image or overlays.
  • Restore an existing VM: replace that VM and its disks with the checkpoint. Current changes are discarded; console sessions disconnect and must reconnect.

More than registers and RAM

Saved CPU state includes general/special registers, extended state, MSRs and CPUID. Interrupt controllers, timers and clock state matter too. Device snapshots preserve virtio registers, queue configuration and packed-ring cursors, plus UART registers. Logical disk contents include CoW, RAM-backed disks and read-only ISO media.

The state file uses gzip-compressed Go gob encoding. Disk assets are stored in a uniquely named sibling directory. Full disk copying takes space and keeps the VM paused during capture: this is neither an incremental snapshot nor live migration.

Save here, start there

Paths belong to the daemon host. Create the checkpoint's parent directory there before saving:

./bin/kube-vmctl pause --uid worker-a
./bin/kube-vmctl resume --uid worker-a

./bin/kube-vmctl save --uid worker-a \
  --path /var/lib/kube-vm/checkpoints/worker.snap
./bin/kube-vmctl snapshot inspect \
  /var/lib/kube-vm/checkpoints/worker.snap

Inspect reports storage_dir, disk sizes and checksums. Securely transfer both worker.snap and that exact storage directory to the destination, preserving their relative placement. Stop the source before running its copy on a shared network. On the destination daemon:

./bin/kube-vmctl start --uid worker-b --detach \
  --snapshot /var/lib/kube-vm/checkpoints/worker.snap

To roll back an existing instance instead, use kube-vmctl restore --uid worker-a --path /path/worker.snap. Save any current work first. Saved CPU, memory and networking settings override ordinary launch defaults.

Checkpoints are sensitive and must be trusted. RAM and disks may contain credentials or application secrets. Files are neither encrypted nor signed. Hashes detect accidental corruption, not authenticity; do not restore an untrusted checkpoint.

Portable does not mean universal

Host compatibility is checked conservatively

Use compatible Linux/amd64 KVM hosts and the supported checkpoint format. CPU execution features, XSAVE layout and KVM capabilities must match the saved requirements. TSC frequency differences need supported KVM scaling. Arbitrary Intel/AMD, CPU-model or kernel changes are not guaranteed; test your recovery hosts.

The network outside the VM does not rewind

TAP/NAT backends are recreated. Saved NAT identity persists even if the restored VM has a different UID. Never run source and restored copy with duplicate identities on a shared network. TAP snapshots can use --tap <destination-interface>, but that does not change their network mode.

Host NAT connection tracking, external TCP peers and shared console sessions are not checkpointed. Connections may need re-establishing. Live VSock and vhost-net checkpoints are unsupported. Old version 1/2 files are inspectable but cannot safely resume because essential state was missing.

Restores keep the saved assets immutable

Each restore creates a private .kube-vm-restore-* directory beside the checkpoint and uses those working disk copies. They hold subsequent guest changes and are not automatically deleted. Re-saving also retains old uniquely named storage directories; remove them only when you know no checkpoint references them.

Capture and restoreDisk materializationKVM compatibility/stateCheckpoint operations