How a pod sees the store¶

nixkube gives a pod a closure: a set of store paths. It can present that closure in three ways. This page lists what each way costs, and which one CSI and NRI use.

The three presentations¶

Bind farm. One read-only bind mount for each store path in the closure. A node's closure has about 2443 paths, so a pod gets about 2443 mounts.

Hardlink tree. One hardlink for each file in the closure, in a directory on the node's disk. A node's closure has about 375,582 files.

composefs. An EROFS image holds the closure's metadata, and an overlay mount reads the file contents from /nix/store. The image is a Nix derivation (nix/composefs.nix), so a node can substitute an image that CI or pynixd built. A pod gets one mount.

Which one a pod gets¶

src/presentation.py renders this chart from the functions that CSI and NRI call, and the presentationChartIsCurrent check fails when the committed chart differs. After you change a decision, run nix run --file . presentation-chart-update.

The chart covers the decisions, not the call sites. The test cells check the call sites: each one asserts how its volumes and /nix mounts are presented.

Comparison¶

	Bind farm	Hardlink tree	composefs
Mounts per pod	One per store path (about 2443)	One	One
Time to build	14 us per path (about 34 ms)	22 us per file (about 8 s)	One Nix build of the image, or a substitution
Disk	Nothing	A directory entry for each file	One image for each closure, shared by all pods
Kernel	`mount_setattr` (Linux 5.12)	Any	EROFS, and overlay data-only lower layers (Linux 6.5)
Limits	`fs.mount-max` is 100,000 for each mount namespace	The tree must be on the same filesystem as the store	None found
Host store on NixOS	Works	Fails: a link out of the host's store gives EXDEV	Works
`hostUsers: false`	Works	Works	CSI: fails, the runtime cannot idmap an overlay. NRI: works, nobody idmaps the mount
Kata and other VM runtimes	Fails: virtio-fs does not serve each bind	Works	Not used
`subPath`	Fails: kubelet's `subPath` bind does not include submounts	Works	Works
Read-write	The gaps between the binds are writable, on the volume's disk	An overlay over the tree, upper directory on the volume's disk	The overlay's upper directory, on the volume's disk
Cleanup	The mounts go with the pod	Delete the tree	Unmount; the image stays until GC collects it

Two more facts apply to the farm and the tree:

A bind and a hardlink share the inode with /nix/store. A write through one changes the node's store. nixkube makes every bind read-only, and gives a read-write tree an overlay. composefs writes go to the overlay's upper directory, so the store is never written.
overlayfs cannot use a bind farm as a lower layer. It does not follow submounts there, and it shows empty directories.

CSI¶

nixkube tries composefs first, and uses a hardlink tree when composefs cannot work. It decides for each volume:

nixkube.csi.composefs = false: a hardlink tree.
The kernel fails the probe (no EROFS, as on Talos 1.11 to 1.13 and GKE COS 125 and 129): a hardlink tree.
The pod has hostUsers: false: a hardlink tree. On a host-store node nixkube refuses the volume, because a hardlink tree cannot work there.
Otherwise: composefs.

The probe mounts a small image and accepts only an exact root. A kernel that silently shows the whole base directory fails it.

CSI does not use the bind farm, because it breaks subPath.

NRI¶

NRI puts /nix into a container after the runtime creates it, so no runtime idmaps it and subPath does not apply. NRI uses the same image and the same kernel probe as CSI.

A VM runtime (Kata): a hardlink tree, served through virtio-fs.
nixkube.nri.composefs is true and the kernel passes the probe: composefs. This includes hostUsers: false pods and host-store nodes.
A host-store node: the bind farm.
Otherwise: the bind farm. NRI_BIND_FARM=false gives a hardlink tree.

The mount worker mounts the image and the overlay in a private mount namespace, and clones the overlay's nix directory into the container. The clone keeps the overlay, and the overlay keeps the image mount, after the worker exits.