Build re-entrancy: delegating the goal system to a real Nix

Sources read, all citations below name the function beside the line number:

Tree

Version

Role

~/Code/nix_r

2.36.0

primary read

~/Code/nix-master

2.35.0

cross-check

/nix/store/2ijv0g60…-source

2.34.8

the floor (nixVersions.nix_2_34.src)

installed nix

2.34.8

what the measurement below ran against

~/Code/nix

351bc2449, 2026-08-03

upstream HEAD, read for Facts 9 to 11

Claims are tagged [measured], [read], or [inferred]. Nothing here is from memory.

The goal

Stop reimplementing Nix's goal graph in pynixd/goals/. Instead hand a real Nix a BuildPathsWithResults request with --max-jobs 0 and a builders entry pointing back at pynixd, so pynixd receives atomic BuildDerivation units it can distribute across a fleet without reasoning about dependencies, substitution, CA resolution or dynamic derivations.

The invariant

Everything below reduces to one rule:

Only the process holding the output-path lock, or a child of it, may write that output path.

pynixd holds no locks of its own, so the hazard exists at exactly one moment: when pynixd forwards a write to a store while an inner Nix holds the lock.

Fact 1 — output locks are held for the whole build hook

DerivationBuildingGoal::tryToBuild (2.36 derivation-building-goal.cc:308) acquires PathLocks on the output paths in its acquireResources lambda (:418) before calling tryBuildHook, and releases them only after the hook exits — success at :804, failure at :754.

The primitive is flock, in lockFile (unix/pathlocks.cc:38) via PathLocks::lockPaths (:71). flock is per open file description, so a second open() conflicts even inside the same process.

[measured] on 2.34.8 with a probe hook that accepts and then tests the lock:

parent_cmd=nix-daemon    euid=0
lockfile=/nix/store/cqal7caw…-reent-probe-….lock
lockfile_exists=yes
RESULT=lock_HELD_by_parent

So on a normal multi-user install the goal system runs inside nix-daemon, which is a LocalStore, and any independent writer of that output path blocks. A naive --builders loop back to the same store deadlocks rather than erroring.

Fact 2 — the lock is conditional on the store type, and upstream says why

[read], identical in 2.34.8 (:382), 2.35.0 (:451), 2.36.0 (:398), all inside DerivationBuildingGoal::tryToBuild:

std::set<std::filesystem::path> lockFiles;
/* FIXME: Should lock something like the drv itself so we don't build same CA drv concurrently */
if (auto * localStore = dynamic_cast<LocalStore *>(&worker.store)) {
    /* If we aren't a local store, we might need to use the local store as
       a build remote, but that would cause a deadlock. */
    /* FIXME: Make it so we can use ourselves as a build remote even if we
       are the local store (separate locking for building vs scheduling? */
    /* FIXME: find some way to lock for scheduling for the other stores so
       a forking daemon with --store still won't farm out redundant builds. */

If the scheduling store is not a LocalStore, lockFiles stays empty and no lock is ever taken. Upstream wrote this comment for precisely our use case.

The stated price is in the third FIXME: no cross-process build dedup. So BuildQueue dedup keyed by .drv stops being an optimisation and becomes load-bearing.

Relevant hierarchy [read]: LocalStore : IndirectRootStore : LocalFSStore (local-store.hh:182), and UDSRemoteStore : IndirectRootStore, RemoteStore (uds-remote-store.hh:62). A daemon-backed store is a LocalFSStore but is not a LocalStore — which is what makes it lock-free as a scheduler, while still satisfying the LocalFSStore casts elsewhere in the goal.

Fact 3 — locksHeld is the sanctioned escape, and it is narrow

LocalStore::locksHeld (local-store.hh:258) is a public PathSet consulted in exactly one place: LocalStore::addToStore (local-store.cc:1075, check at :1092) skips PathLocks for paths listed there. nix __build-remote sets it before copying outputs back (build-remote.cc:417-419, commented /* FIXME: ugly */).

The other locking write path does not honour it: LocalStore::addToStoreFromDump (local-store.cc:1192) locks unconditionally at :1313. So the only safe in-lock ingest call is addToStore(ValidPathInfo, Source).

Fact 4 — running the goal machinery over a non-LocalStore needs the C++ API

No CLI can do it, in any version [read]:

  • 2.36: daemon::processConnection takes the builder from store->getBuilder() (daemon.cc:1136); RemoteStore::getBuilder returns a forwarding RemoteBuilder (remote-store.cc:741). Store::buildPaths no longer exists.

  • 2.34: Store::buildPathsWithResults is virtual with a Worker-based default (entry-points.cc:48) and RemoteStore overrides it (remote-store.hh:113).

So nix daemon --stdio --store daemon relays builds upstream and runs no goals; the upstream daemon's own max-jobs/builders decide. That topology is a trap.

The C++ entry points, which nanopynix can reach:

Version

Call

2.36

make_ref<LocalBuilder>(store, evalStore)->buildPathsWithResults(reqs, mode) — Store::getBuilder default (store-api.cc:151), LocalBuilder in build/worker.hh:80

2.34

qualified call bypassing virtual dispatch: store->Store::buildPathsWithResults(reqs, mode, evalStore)

LocalBuilder::getWorker (worker.hh:96) constructs a fresh Worker per call, commented "to avoid reusing a worker between calls, allowing for thread safety" — so concurrent pynixd requests are naturally isolated.

Caveat [read]: Worker::Worker binds settings(nix::settings.getWorkerSettings()) (worker.cc:30), so max-jobs and builders are process-global, not per-request. In-process scheduling cannot vary them per client.

Fact 5 — prior art: Nix already does this to itself

Recursive Nix serves a nested daemon connection over the same store while the parent Worker is blocked, using a fresh Worker (derivation-building-goal.cc:905-917):

"We create a fresh Worker here because the parent Worker is blocked waiting for the current build to finish… Ideally we should reuse the same Worker to share scheduling state."

Two Workers over one store in one process is a supported shape, and upstream's own answer to sharing scheduling state is: don't.

Fact 6 — the build hook protocol

[measured] — a shell script implementing this got accepted and logged building '…drv' on 'probe://fake'. Layout from HookInstance::HookInstance (unix/build/hook-instance.cc:36-88):

Channel

Direction

Content

stdin (fd 0)

in

settings (1,name,value… terminated by 0), then per attempt: try, amWilling, system, drvPath, requiredFeatures; after accept: inputPaths, wantedOutputs

stderr (fd 2)

out

control: # accept / # decline / # decline-permanently / # postpone, then one line of machine name; other lines are relayed as log

fd 4

out

build log

fd 5

in

read side of the log pipe (hack for ssh stderr)

Selection in DerivationBuildingGoal::tryBuildHook (:1165) requires only that build-hook is non-empty and the drv is valid at :1172 — builders is not consulted, so a custom hook needs no machines file. amWilling is getNrLocalBuilds() < maxBuildJobs (:1182), i.e. 0 under --max-jobs 0.

After the hook exits, DerivationBuildingGoal::buildWithHook (:591) re-checks validity and throws "some outputs are unexpectedly invalid" (:775-777) if the outputs are not registered. That message is the failure mode for every "ingest didn't land" bug.

Fact 7 — what stock nix __build-remote does

main_build_remote (src/nix/build-remote/build-remote.cc):

  • :331 — if the remote reports untrusted and the drv is not CA, it falls back to copyClosure + buildPathsWithResults at :378. pynixd must report trusted in the handshake (RemoteStore::isTrustedClient, remote-store.cc:889, reads conn->remoteTrustsUs) or the hook recurses into pynixd's own scheduler. This is the single sharpest trap in the stock path.

  • :337-362 — the BasicDerivation sent to us has its inputs replaced by the full input closure. It is self-contained; we never read the .drv.

  • :309 / :420 — inputs copied in, outputs copied back, both NoCheckSigs.

  • :389-405 — for CA, every wanted output must appear in builtOutputs or the assert at :399 aborts the hook; realisations then registered at :427.

  • :281 — one exclusive <uri>.upload-lock per builder URI, serialising input copies across concurrent hooks. openSlotLock (:42) in <stateDir>/current-load is the only concurrency cap, so the machine's maxJobs field governs how many hooks exist at once.

  • Machine::systemSupported (machines.cc:39) returns true for "builtin" unconditionally — so builtin:fetchurl derivations are shipped to us too.

  • Feature routing is Machine::allSupported (machines.cc:44); with --max-jobs 0 there is no local fallback, so a missing feature is a hard failure.

Daemon-side, BuildDerivation refuses input-addressed drvs from untrusted clients (daemon.cc:625-676, long comment on the trust model).

Fact 8 — external-builders: the minimal-ingest option

Present as far back as 2.34 (local-settings.hh:652, unix/build/external-derivation-builder.cc), experimental feature external-builders.

ExternalDerivationBuilder::startChild writes a JSON file with builder, args, env, inputPaths, outputs, realStoreDir, storeDir, system, tmpDir, and execs our program with its path. We materialise the output directories and exit; Nix does registration, reference scanning, canonicalisation, CA hashing and realisation creation itself, because the class inherits DerivationBuilderImpl. stdout/stderr are already the build-log pty (openSlave()). It runs as the build user (setUser()), unsandboxed.

Three structural catches:

  1. Requires a LocalStore — it is selected inside the local-build branch of DerivationBuildingGoal::tryToBuild (:344-382), after the dynamic_cast<LocalStore *> at :347.

  2. It is chosen before the maxJobsZero / platform / feature checks (if (ext) return LocalBuildCapability{...} at :357), so it silently overrides --max-jobs 0 and foreign-platform rejection. But DerivationBuildingGoal::buildLocally then gates on curBuilds >= maxBuildJobs (:857) — so with max-jobs 0 the goal parks on waitForBuildSlot for a slot that never opens. [inferred, not measured] Concurrency is therefore max-jobs, and each build also consumes a nixbld user.

  3. Dispatch is by drv.platform alone (LocalSettings::findExternalDerivationBuilderIfSupported, globals.cc:283); the JSON carries no drv path and no requiredSystemFeatures. Fleet dedup cannot be keyed on the .drv, and for floating CA the outputs map holds scratch paths, not final ones.

The hook is tried before the local build unless preferLocalBuild (:568-583), so the two compose: hook as the routed lane, external builder as the fallback lane.

The JSON stays at "version": 1 at HEAD [read] (local-settings.hh:711). It carries no RPC. Fact 9 is the separate feature that does.

Fact 9 — builder-rpc-v0 inverts the ingest direction

New since the rest of this note. 55eea4554, Artemis Tosini, 2025-11-17, "Implement new builder-rpc-v0 derivation feature". It is in HEAD [read]. The manual section is doc/manual/source/store/building.md:225-270.

A derivation asks for it by putting builder-rpc-v0 in requiredSystemFeatures. Nix then gives the builder a restricted daemon socket and no output paths in the environment. The builder submits each output itself, with nix store add and nix store submit-output, or by speaking the protocol.

Three properties matter to pynixd:

  • Every output is content-addressed. The IPC path supports no input-addressed output.

  • Order is the contract. Reference scanning covers the inputs and every store object added so far, so the builder must add foo before a bar that refers to it. Nix returns each computed store path in the response.

  • Self-references are not supported yet, and the manual says why: only rewriting can resolve them, and the point of the IPC path is to stop Nix rewriting opaque data.

Two new wire operations carry it (worker-protocol.hh:275-279): SubmitOutput = 1000 and AddToStoreScanning = 1001. Both sit above the normal opcode range on purpose. A new serialiser for SingleDerivedPath arrives with them (:349). SubmitStore (submit-store.hh) is the store interface, and nix store submit-output (src/nix/store-submit-output.cc) is the command, gated on Xp::DynamicDerivations.

DerivationBuildingGoal::buildLocally now passes a daemon::RecursiveFlag into processDaemonConnection instead of the constant daemon::Recursive, so the restricted daemon serves two different surfaces from one call site.

This is a second re-entrancy direction, and it points the other way. The rest of this note is about pynixd calling out to a Nix goal system. This one is a build calling in to a daemon that pynixd may be proxying. A pynixd that proxies a builder-rpc-v0 build must pass these operations through.

Fact 10 — the protocol version is a pair now, and pynixd models it as an int

[read] worker-protocol.cc:20-54. WorkerProto::Version is {number: {major, minor}, features: set<string>}. operator<=> (:57) compares the number and then requires the feature set to be a subset, so it returns partial_ordering and two versions can be unordered.

The three constants:

Constant

Number

Features

latest

1.38

realisation-with-path-not-hash, delete-dead-specific-referrers

minimum

1.18

none

builderRpcV0

1.38

realisation-with-path-not-hash, disable-set-options, add-to-store-scanning, submit-output

Read again at origin/master 4401a297c, 2026-08-16. This table gave latest a third feature, build-result-memory. That name is in no file of src/, and worker-protocol.cc:26-30 gives latest the two above. The correction is here rather than in a new fact, because a wrong row is worse than a missing one.

builderRpcV0 is pinned, and the comment at worker-protocol.hh:122-125 says why: "Should never change, as any modification would be derivation-visible."

builderRpcV0 is not the recursive-nix connection. It serves the builder-rpc-v0 derivation feature of Fact 9, and that surface starts no build. processDaemonConnection takes a daemon::RecursiveFlag and serves both from one call site, which is what makes the two easy to confuse. Nix's own comment at worker-protocol.hh:140 names recursive-nix for disable-set-options, and both surfaces take that feature for the same reason: a builder cannot change the settings of the daemon that serves it.

The five feature names are at worker-protocol.hh:132-152.

nix-daemon-protocol holds PROTOCOL_VERSION = proto(1, 38) as one integer (constants.py:20), and SUPPORTED_PROTOCOL_VERSIONS is a range of minors (:22). A bare 1.38 is no longer the whole handshake. Two peers can both say 1.38 and still disagree on submit-output. The AGENTS.md line "pynixd will advertise 1.38 support" is now under-specified.

Fact 11 — upstream started the split this note asks for

[read] 4125ece6c, John Ericson, 2026-03-20, "Separate building/scheduling from storage". It is in HEAD. The message calls it "the major first step of #5025".

It pulls the build methods out of Store into a new Builder class, and lets some stores give a Builder. Worker implements Builder, so a Worker can own or borrow stores and be one scheduler, rather than each Store method making a Worker behind the scenes.

Issue #5025 is "use the store interface for --builders". Its end state deletes the build hook: a --builders entry becomes a plain Store that the local Worker drives. That is the seam this whole note is trying to build by hand.

Fact 4 records the consequence that already landed: 2.36 takes the builder from store->getBuilder() and Store::buildPaths is gone.

Ruled out

  • local-overlay store as a way to give the inner Nix a harmless place to write: the lower store directory "must not change at all" while mounted (local-overlay-store.md), which a live cache violates by definition.

  • nix daemon --stdio --store daemon: relays, runs no goals (Fact 4).

  • pynixd as a substituter instead of a builder: re-entrancy evaporates since substitution holds no long-lived lock, but queryPathInfo becomes a synchronous fleet build and we lose BuildResult and build logs.

The decision that collapses the option matrix

Does pynixd run as root?

  • Root — pynixd can hold its own LocalStore handle, seed locksHeld, and write directly. Every option is open, including external-builders.

  • Not root — writes go through the real daemon, which takes the lock (local-store.cc:1085). The scheduling store must not be a LocalStore, and external-builders is unavailable.

Topologies

Scheduler runs on

Locks held

Who ingests

Needs

A

spawned nix daemon --stdio, LocalStore

yes

the hook must; pynixd must not register outputs

root

B

in-process Worker over UDSRemoteStore

no

either

nanopynix

C

LocalStore + external-builders

yes (we are the child)

Nix itself

root, Xp feature

Under A, pynixd's nested endpoint degenerates to a virtual builder that proxies one build to a chosen fleet machine — BuildDerivation, QueryValidPaths, QueryPathInfo, NarFromPath — and copyPaths (build-remote.cc:420) populates the cache for us. The OutputPuller in architecture.md disappears.

Under B, no lock is ever held, so pynixd may register outputs itself and the existing puller stays. This is the topology nanopynix unlocks and it is the only one that works unprivileged.

Open questions

  1. Custom hook vs stock nix __build-remote. A custom build-hook drops the daemon-protocol callback, the trust negotiation, the upload lock and the slot locks, and puts ingest in a process that is a child of the lock holder by construction. Cost: we own reference metadata, CA realisation registration, and postpone/decline backpressure — the last of which is also the only real queueing primitive Nix offers a hook.

  2. Fix the FIXME upstream. derivation-building-goal.cc:399-406 asks for "separate locking for building vs scheduling" and names our deadlock. A scheduler-only store param, or exposing locksHeld as a store setting, would make the topologies above stop being workarounds.

    Partly answered: see Fact 11. Upstream is doing the larger version of this in #5025, and the first step is in HEAD. The remaining question is whether to wait, or to add the narrow scheduler-only param now. Waiting costs nothing if topology B is correct, because topology B takes no lock in the first place. Decide this after the Fact 2 measurement.

  3. Stale negative path-info cache. Store::queryPathInfo (store-api.cc:636) caches negatives for narinfo-cache-negative-ttl (default 3600s); Store::isValidPath (:541) reads that cache but never writes negatives to it. Whether a scheduler can observe a stale negative and trip "some outputs are unexpectedly invalid" is unverified. Cheap insurance: path-info-cache-size = 0 on the scheduling store — which is what runDaemon already does for daemons (nix/unix/daemon.cc:483).

  4. Suspected inversion in 2.36 nix daemon --stdio. nix/unix/daemon.cc:497 reads processOps |= !forceTrustClientOpt || *forceTrustClientOpt != NotTrusted; while the comment two lines above says the opposite — as written, --force-untrusted is the only case that forwards. Not verified against upstream master; pass --process-ops explicitly regardless.

  5. Model the feature set in nix-daemon-protocol. Fact 10 shows one integer is no longer the version. This is independent of the delegation decision: a pynixd that proxies any recent Nix needs it either way, and it is a prerequisite for proxying a builder-rpc-v0 build. Size it first — Version becomes a pair, and SUPPORTED_PROTOCOL_VERSIONS becomes a negotiation rather than a membership test.

Reproducing the measurement

The probe hook lives in this session's scratchpad; it is ~20 lines of bash that writes # accept\n<name>\n to stderr, sleeps, then flock -n tests <out>.lock. Run it against a private chroot store, never the machine's shared daemon — a --builders loop back to the same store wedges the daemon instead of erroring.