03 / Storage architecture

One base. Private changes. A complete disk.

The guest sees a virtio block device. Behind it, kube-vm can combine a read-only base with a writable copy-on-write overlay—without teaching the guest about layers.

Read the latest cluster, wherever it lives

Disk offsets are split into 4 KiB clusters by default. An in-memory index records the latest overlay record for each changed cluster. Reads consult that index; writes append a new version instead of changing the base.

Copy-on-write cluster lookup and append An unchanged read goes to the base disk. A partial write merges with current cluster data, appends an overlay record, and updates the index. Later reads use the overlay. virtio-blklogical disk offsets Cluster indexcluster → latest record Immutable basefallback on a miss Private overlay append → append → write: merge, then append read miss
  1. There is no index entry for this cluster. Read the requested bytes from the base at the same logical offset.
  2. Read the current cluster from the latest overlay record or the base. Replace only the byte range the guest changed; preserve the rest.
  3. Write the cluster number and complete cluster data to the end of the overlay. Point the in-memory index at this newest record.
  4. The index now hits. Read the requested bytes from the new overlay record; the original base still has its original contents.
Amber = base reads; blue = overlay traffic; green = guest/index path. This is kube-vm's own CoW format, not qcow2 or a raw ext4 overlay.

What crosses the virtio queue?

A block request has a 16-byte header, one or more data buffers, and a one-byte status. Its sector field uses 512-byte units, independently of the overlay's cluster size. A filesystem block is yet another layer; do not assume all three sizes are identical.

  • Read: fill guest-writable data buffers from the backing store.
  • Write: consume guest-provided data buffers and update the backing store.
  • Flush: ask storage to synchronize, rather than transferring a payload.
  • Get ID / unsupported requests: return the supported identification response or an unsupported status.

The device checks request layout, buffer direction and capacity bounds. It returns OK, I/O error or unsupported status, publishes the used descriptor chain, and notifies the guest as required by the queue's event state. A read-only device advertises that property and rejects writes.

The overlay is a log, not a second filesystem

The header stores a format magic, version, cluster size and logical disk size. Records contain a cluster number followed by that cluster's bytes. Opening an existing overlay scans its records to rebuild the latest-record index.

Repeated writes to the same cluster append repeated versions. This saves the cost of copying an entire base for each VM, but does not mean overlay growth is bounded by the amount of unique changed data. A small guest write can cause a full-cluster read/merge/append.

Keep the pair together. Ordinary CoW operation still depends on the unchanged base. Do not modify the base under an overlay, share a writable overlay between running VMs, or run an ext4 checker on the overlay container. A full checkpoint instead materializes the logical disk contents into portable assets.

Other backing stores

Block specifications also support local files, read-only HTTP(S) storage, and mem:BYTES RAM disks. A plain writable -block image changes in place; append :ro for read-only file/memory specs. CoW explicitly separates the shared base from each VM's writes.

Give each VM its own overlay

With a daemon running, attach a base and a private overlay using the convenience flag:

./bin/kube-vmctl start --uid worker-a --detach \
  --kernel ./bzImage \
  --cmdline "console=hvc0 root=/dev/vda rw" \
  --cow-block ./disks/base.img=./disks/worker-a.overlay

The canonical block specification is cow:/path/base.img?overlay=/path/vm.overlay. The CLI encodes --cow-block BASE=OVERLAY into that form. Booting without an initrd requires the root filesystem and block drivers to be built into the kernel.

Measure guest-visible I/O

The API and dashboard report per-disk operations, transferred bytes, errors, in-flight work and completion latency. Those metrics describe guest requests, not host physical I/O or guest filesystem free space; CoW write amplification is not included.

Block requestsCoW format and I/OBlock specifications