The modes, compared¶

pydd builds one Python environment in nine modes. The plan is the same in each mode. A mode decides what the planner may do, how the plan becomes derivations, and how a source distribution is built.

Modes that build a source node the same way give the same environment, and a test proves it. nix run --file . test-projects builds each of the 19 test projects in every mode, in two groups. It fails when two modes of a group give different store paths for one environment.

The plain group: cli, huggorm, drv, recursive, recursive-eval and out. A source node runs pydd's own build scripts.
The stdenv group: recursive-nixpkgs, stdenv and out-stdenv. A source node is the derivation stdenv.mkDerivation writes, and stdenv strips and patches what it builds, so these environments differ from the plain ones.

The levels of restriction¶

This repository exists to show what a planner can do at each level. From the most restricted to the least:

level	Nix	a planner may	modes
`$out`	any release	write one derivation, its root, as ATerm to `$out`	`out`, `out-stdenv`
`builder-rpc-v0`	master	add derivations through a socket, and submit the root	`cli`, `huggorm`, `drv`, `stdenv`
`builder-rpc-v0` with EnsurePath	master with Lillecarl/nix#371	also `builtins.storePath` and `builtins.appendContext`	none built (pydd#8); `recursive-eval` shows what it allows
recursive-nix	any release, with `recursive-nix`	also `QueryPathInfo`, and building	`recursive`, `recursive-eval`, `recursive-nixpkgs`

pydd uses recursive-nix as a builder-rpc-v0 with more allowed, and never builds through it. recursive-nix lets a build build, and a build that waits on another build holds a build slot. So a pydd planner only adds derivations and evaluates. Every planner sets allow-import-from-derivation = false, so an evaluator in it cannot build to import. .override { allowImportFromDerivation = true; } on the root, or overrideAttrs on mkPlanner's result, changes that.

On a release, recursive-nix gives the whole graph to the planner. A planner writes every node through the socket, and its root to $out. Nix lets an output name a path that the build added. So the host makes no node, as with builder-rpc-v0, and the environments are the same paths.

The Nix: `builder-rpc-v0`, recursive-nix, or `$out`¶

builder-rpc-v0 is on Nix master, and in no release. Nix gives the planner a restricted daemon socket. The planner writes each derivation of its graph through that socket, and then submits the root as its output.

recursive-nix is in every release, as an experimental feature. The planner gets a daemon socket, as above. A release has no SubmitOutput, so the planner writes its root as ATerm to $out.

On a release with neither, a planner writes one derivation: its root. It writes the root as ATerm text to $out, and Nix reads that text as a derivation. The planner cannot add any other derivation, so the host makes every other node before the planner runs.

The writer: `nix`, huggorm, or `pydd-drv`¶

cli is the stock nix binary. The planner runs nix derivation add once for each derivation, and nix store submit-output for the root.
huggorm is Python bindings to Nix's own C++ libraries, libnixstore and libnixexpr. It is not a second implementation of Nix: the code that runs is Nix's. The planner writes in process, and it can evaluate Nix code: this mode evaluates lib/python/graph.nix.
drv is pydd-drv, a small C++ tool over libnixstore (tools/pydd-drv, Lillecarl/pydd#1). It is the specialised CLI. add and submit do what the two nix commands do, and graph writes a whole graph of JSON derivations in one process.
out writes through huggorm today. huggorm stages each derivation in an in-memory store, and the planner copies the root's ATerm to $out. pydd-drv graph --out does the same for a planner in any language (examples/graph). pydd's own planners cannot use it yet, because they add one derivation per call, and a new process does not keep the in-memory store (Lillecarl/pydd#15).

The nine modes¶

mode	Nix	writer	plan to derivations	a source node
`cli`	master, `builder-rpc-v0`	`nix`	`pydd.modes.render`, in Python	pydd's scripts
`huggorm`	master, `builder-rpc-v0`	huggorm	`graph.nix`, in huggorm's evaluator	pydd's scripts
`drv`	master, `builder-rpc-v0`	`pydd-drv`	`pydd.modes.render`, in Python	pydd's scripts
`stdenv`	master, `builder-rpc-v0`	`nix`	`pydd.modes.render`, in Python	stdenv, as data
`recursive`	2.35, recursive-nix	`nix`	`pydd.modes.render`, in Python	pydd's scripts
`recursive-eval`	2.35, recursive-nix	`nix`	`graph.nix`, in `nix eval`	pydd's scripts
`recursive-nixpkgs`	2.35, recursive-nix	`nix`	`graph.nix`, in `nix eval`	stdenv, from nixpkgs in the planner
`out`	2.35, `$out`	huggorm	`pydd.modes.render`, in Python	pydd's scripts
`out-stdenv`	2.35, `$out`	huggorm	`pydd.modes.render`, in Python	stdenv, as data

recursive-eval is what EnsurePath buys. A stock nix eval in the planner evaluates graph.nix over lib/modes/nix-eval.nix, which attaches the context with builtins.storePath and builtins.appendContext. It needs no huggorm and no renderer, and it writes the same .drv for every node kind (checks.tests). On builder-rpc-v0 the same needs Lillecarl/nix#371.

Three ways to give a node stdenv¶

A node of pydd runs no stdenv, so pydd does stdenv's work by hand: -g0, CPPFLAGS and LDFLAGS, and scripts/fixup-elf.py in place of autoPatchelfHook. Three ways give a node stdenv instead:

	nixpkgs in the planner	stdenv as data	host builders
modes	`recursive-nixpkgs`	`stdenv`, `out-stdenv`, any other	any (`builders`, pydd#6)
how	the planner imports nixpkgs and calls `stdenv.mkDerivation`	the host evaluates `stdenv.mkDerivation` once, and the planner lays a node's attributes over the result	the host calls a function per artefact, before planning
needs	recursive-nix	nothing more	nothing more
keeps	all of nixpkgs, at plan time	stdenv's phases and setup hooks	all of nixpkgs
loses	overlays and config, except as files and JSON	`mkDerivation`'s logic at evaluation: `overrideAttrs`, `passthru`, inputs it derives	the planner's choice: the host decides per artefact

The first two write the same derivation. checks.stdenvAsData holds them to one .drv for every source node, and test-projects to one environment for every project. A planner with no nixpkgs, on builder-rpc-v0 or on a release with no socket, so writes exactly what a planner that imports nixpkgs writes.

nixpkgs is evaluated only where a node forces it. A wheel node never does, so a planner of a wheel-only lock never imports nixpkgs. Each sdist planner and local-project planner that builds from source evaluates stdenv once.

Warning

An overlay does not reach a planner. An overlay is a Nix function, and no function crosses into a sandbox. recursive-nixpkgs takes overlays only as files (plannerNixpkgsOverlays) and config only as JSON (plannerNixpkgsConfig). A package set that the caller changed in memory is not the one the planner imports.

"${pkgs.path}" copies nixpkgs a second time, because a path value is copied on interpolation even when it lies in the store. pydd names the path with builtins.storePath instead, in an impure evaluation.

How `out` builds a graph from one derivation¶

The host is the evaluation that the user starts, such as nix build --file . env. It runs before any build, outside every builder, the way the evaluation of a nixpkgs package does. It instantiates derivations: it writes each .drv to the store, and builds nothing. A planner is a builder, which runs later, in a sandbox.

lib/modes/out.nix makes the host's nodes. It evaluates graph.nix for each one, takes its .drv path with unsafeDiscardOutputDependency, and passes the list to every planner as hostNodes. Each .drv is then an input of the planner, present in its sandbox, while no output of it is built.

A planner on a release writes one derivation, and that derivation may name others that already exist. A root names its inputs by .drv path. Nix accepts any input that is in the store, and the planner's sandbox holds every input of the planner. So the host makes each .drv that a root could need before the planner runs, and the planner decides which of them the root names. Three things make that work:

The host makes a node for every wheel of the lock, for every platform, with the same graph.nix and the same document that the planner would write. So the host's .drv and the planner's are one path. The host only instantiates them: Nix builds the ones that the root names, and no other one.
A node that only a build can shape gets a planner of its own. The backend of an sdist is declared inside the archive, and no evaluation has read the archive. The host makes one planner per sdist, one per local project, and one for the environment under an application. Each writes one derivation, its node, as its own $out. The root names that node as a dynamic output (builtins.outputOf), a node of kind planned. So one node per package holds: each package's derivation is either one the host made, or the one derivation of its own planner.
The planner computes every path exactly, and writes only the root. huggorm stages each derivation in an in-memory store, so the paths match the other modes byte for byte. The planner then copies the root's ATerm to $out. Nix registers no reference to a path that the sandbox lacks, and the root then silently becomes another derivation. So the writer refuses a root that names such a path.

So out chooses at build time, and it does not construct. Its planners run in builders, and make each decision that needs a build: which wheel fits the target, and what an sdist's archive declares. Every node they name is one the host made before. That suffices when the host can list the candidates, as one node per package can. It fails when the number or the shape of the nodes is known only after a build, such as a node per source file of an archive, or nesting deeper than the host prepared (sdistPlannerDepth). builder-rpc-v0 constructs: a planner writes any derivation it decides on.

The cost is on the host. It instantiates a node for every wheel it might need, and a planner wherever only a build can decide. With builder-rpc-v0 a planner writes what it chose, and nothing more.

"Plan to derivations" is a third, smaller choice. graph.nix says how a node is built. pydd.modes.render makes the same derivations in Python, for a planner with no evaluator. tests/test_render.py holds the two to the same .drv for every node kind.

Measured¶

Measured at a56e8569 on 2026-10-13, x86_64, 16 cores; the median of 3 runs.

	cli	huggorm	drv	recursive	recursive-eval	recursive-nixpkgs	out	stdenv	out-stdenv
planner closure (MiB)	322.7	509.5	305.1	322.2	322.2	322.2	508.5	322.7	508.5
lines of the mode	399	204	672	399	148	148	631	444	676
a build that does nothing (s)	1.49	1.61	1.73	1.39	1.53	1.42	1.75	2.56	2.60

Planner time, in seconds:

project	packages	cli	huggorm	drv	recursive	recursive-eval	recursive-nixpkgs	out	stdenv	out-stdenv
application	4	3.88	3.28	2.79	2.98	2.88	3.33	10.67	3.20	9.87
binary-app	1	2.90	2.84	3.24	2.84	2.83	2.88	2.98	2.97	2.84
binary-nixpkgs	1	3.17	2.71	2.88	3.12	2.65	3.30	3.12	2.85	3.10
git-app	0	2.81	2.56	2.44	2.95	2.69	3.26	6.94	3.03	6.58
local-index	2	6.71	7.20	6.44	6.99	6.72	6.93	6.69	6.62	6.87
native-app	3	6.26	7.25	6.43	6.33	6.61	7.10	6.63	6.97	6.96
native-host	3	6.29	6.20	6.06	7.61	7.22	7.08	6.61	6.62	6.92
numpy-sdist	1	7.22	6.40	6.46	7.19	6.44	7.11	7.01	6.80	7.09
packaged-app	4	6.45	6.29	5.86	6.96	6.13	7.37	6.73	6.53	6.55
packaged-app-bin	4	7.18	8.15	7.01	7.95	7.47	7.76	9.95	7.89	10.60
packaged-sdist	4	6.69	6.39	6.38	7.57	6.18	7.46	9.90	6.89	12.07
pure-app	4	4.71	5.41	5.47	5.71	5.48	6.18	4.79	4.39	4.00
root-derivation	4	3.86	3.57	2.98	3.15	2.58	2.77	2.93	2.75	2.94
vendored-app	2	7.29	6.67	6.14	6.81	6.45	6.91	6.78	6.69	6.35
workspace-all	4	3.44	2.95	2.67	3.17	2.77	3.17	8.03	3.12	7.85
workspace-core	0	2.91	3.03	2.76	3.06	2.75	3.23	6.64	3.20	6.92
workspace-core-sdist	4	4.66	3.15	2.69	3.36	2.77	3.35	10.32	3.02	9.80
zmq-sdist	1	7.43	6.77	6.93	6.81	6.57	7.17	6.69	6.96	6.27
zmq-sdist-transitive	1	7.10	6.43	6.10	6.99	6.93	7.25	6.85	7.30	7.32

tools/measure-modes.py measures this, after test-projects built every node. It gives every planner a nonce, so Nix runs each planner of an environment again, nested ones included, and builds no node. So a time is all the planning of one environment.

At this scale the mode decides little, except on a release with no socket. A build that does nothing takes 1.4 to 2.6 s. Each mode with a socket, builder-rpc-v0 or recursive-nix, is within about a second of the others on every project. out and out-stdenv take 6 to 12 s where a project has a local project under an application or a workspace, against about 3 s elsewhere: the host-made planner of the local project runs before the planner of the environment.

nixpkgs in the planner costs little. Measured alone, the sdist planner of psutil in native-app takes 3.19 s in recursive-nixpkgs, where it imports nixpkgs and writes a stdenv.mkDerivation node, against 2.87 s in recursive-eval. A planner that writes no source node never imports nixpkgs, so a wheel-only lock pays nothing.

The closure follows the writer. A planner that writes through huggorm (huggorm, out, out-stdenv) carries about 509 MiB; one that writes through nix or pydd-drv carries 305 to 323 MiB.

The test projects lock at most a few packages. The ddrn lab measured lock files of up to 1911 packages (PACKAGING-FACTS.md), and a large lock is where the modes can differ: one process per node in cli and drv, one evaluator in huggorm, and host evaluation of every wheel in out. That measurement is still to do.

Planner closure is what one planner needs in its sandbox: an interpreter, pydd, and the tool that writes.
Lines of the mode count the files that only that mode needs, beside the core and the Python layer that every mode shares.

What each mode cannot do¶

cli and drv run no evaluator in the planner. So graph.nix exists twice: in Nix, and as pydd.modes.render in Python. tests/test_render.py holds the two to the same .drv for every node kind.

huggorm cannot call builtins.storePath or builtins.appendContext. Both call EnsurePath, and builder-rpc-v0 refuses it. huggorm attaches the string context from Python instead (pydd.modes.evaluate._document.document_value). A plain nix eval in the planner has no such way.

No builder-rpc-v0 mode can evaluate nixpkgs in a planner. Measured in Lillecarl/pydd#9: a planner that imports nixpkgs fails after 33 s with Operation 26 not allowed inside derivation. Operation 26 is QueryPathInfo. fetchToStore calls it after it copies a path literal into the store, and nixpkgs has a path literal in every setup hook. Lillecarl/nix#371 allows EnsurePath, and not QueryPathInfo. recursive-nix allows both, so recursive-nixpkgs works. Stdenv as data needs neither, so stdenv and out-stdenv get stdenv's nodes without it.

The recursive modes could build, and refuse to. recursive-nix gives a planner a socket that builds. pydd's planners only add derivations, and an evaluator in them refuses import from derivation. Nothing in Nix holds a planner to that: it is pydd's convention, and the one setting.

out makes the host do the planning that a planner cannot.

The host makes a node for every wheel of the lock, for every platform, because only the planner knows which wheel it chooses.
An sdist, and a local project, become planners that the host made. A planner reads the output of another through builtins.outputOf.
A root that names a path the sandbox does not hold silently becomes another derivation. OutWriter and pydd-drv graph --out refuse such a root.

What this asks of Nix¶

Each item is a change to Nix, and each one is measured above or in the issue it names.

EnsurePath in builder-rpc-v0 (Lillecarl/nix#371). A planner can then run a plain nix eval over graph.nix, with no huggorm. recursive-eval shows it today, on a release, through recursive-nix.
QueryPathInfo in builder-rpc-v0, or a fetchToStore that does not query. A planner can then evaluate nixpkgs, as recursive-nixpkgs does through recursive-nix. It is the weakest request here: stdenv as data writes the same nodes with no nixpkgs in the planner, at every level.
Tools over the daemon protocol, and not a second protocol. pydd-drv (pydd#1) is the shape: add and submit are the two commands Nix has, and graph writes a whole graph in one process. It computes each placeholder and writes ATerm to $out on a release. Its closure is not much smaller than nix's, because libnixstore is most of both. A subcommand of Nix would cost nothing extra.
Nothing else. One derivation per package works on released Nix today, through host-made nodes. The cost is host evaluation, and the table above shows what the planners cost.