# A third ecosystem: npm

The npm lane plans a project from its `package-lock.json`. Like the Rust
lane, it uses only `pydd.core` and the root wrapper `lib/core/root.nix`.

    root = pydd.node.npm { project = ./.; };

The result is an ordinary derivation. It holds the project in
`lib/node_modules/<name>` and its programs in `bin`. It takes `override`
over the arguments, `overrideAttrs` and the install hooks.

## The lock is the plan

A Python lock says nothing of how to build a package, and a Cargo lock
says nothing of units. An npm lock of version 2 or 3 says almost
everything. Its `packages` map holds one entry for each path that npm
installs, with the URL, the hash, the platform (`os`, `cpu`, `libc`),
`dev`, `optional`, `bin` and `hasInstallScript`.

So the planner runs no npm. It reads the lock in the build, and decides
three things for the platform of the build:

- **Which packages install.** An optional package whose `os`, `cpu` or
  `libc` does not fit is left out, as npm leaves it out. esbuild has one
  such package for each platform.
- **What each package can `require`.** Node looks for
  `node_modules/<name>` in the directory of the package, then in each
  parent directory. The planner walks the lock's paths the same way, and
  a workspace link resolves to its member.
- **What the result leaves out.** `omit` is npm's `--omit`, `[ "dev" ]`
  by default.

The host evaluation never reads the lock. Its cost does not grow with the
number of packages.

## Other locks: pnpm, bun and Yarn v1

The planner also reads `pnpm-lock.yaml`, `bun.lock` and Yarn v1's
`yarn.lock`. A reader in `pydd.node._readers` turns each of them into
npm's map of install paths. The rest of the lane does not change: the
same nodes, the same fetches, the same root. By default the planner reads
the first lock of `package-lock.json`, `pnpm-lock.yaml`, `bun.lock` and
`yarn.lock` that the project holds. `lockfile` chooses another.

All three hash a package as npm does: the `integrity` of its tarball,
which a fixed-output fetch takes as it is.

| lock | install paths | programs | platforms | install scripts |
| --- | --- | --- | --- | --- |
| npm | in the lock | in the lock | in the lock | in the lock |
| bun | in the lock | in the lock | in the lock | those that `trustedDependencies` names |
| pnpm 9 to 11 | a graph; the reader places it | only "has programs" | in the lock | those that `allowBuilds` or `onlyBuiltDependencies` names |
| Yarn v1 | a graph; the reader places it | not recorded | not recorded | not recorded |

**The reader places a graph as npm does.** Each version goes to the top
`node_modules`, unless another version of that name is visible from its
dependent. Then it goes inside the dependent. For every project of
`tests/node`, each tool's lock gives the layout of npm's lock
(`tests/test_readers.py`).

**A node reads what the lock omits.** When a lock does not name a
package's programs, the node reads them from the package's own
`package.json`, and makes the `.bin` links at build time.

**Install scripts follow the tool.** pnpm 10 and bun run the scripts of
the packages that the project allows, and no others. The reader does the
same. pnpm 9 ran every script, so a pnpm 9 project names its scripts, as a
Yarn project does. Yarn v1 runs every script, and its lock does not say which packages
have one. So a Yarn project names them: `installScripts = [ "better-sqlite3" ];`,
or `"pydd": { "installScripts": [...] }` in `package.json`. bun also has a
built-in list of packages that it trusts. pydd does not copy that list.

**Yarn v1 records no platform.** Every package of esbuild, for each
platform, installs for the build. The result omits them as development
dependencies, or keeps them all.

**Yarn 2 and newer are refused.** Their `checksum` is the hash of Yarn's
own zip of the package, not of the tarball. Nix cannot check a fetch
against it. This is the problem of Go's `go.sum` (pydd#25).

**`out` reads npm's lock only.** In `out` the host makes every node, in
Nix. Nix reads no YAML, and the host would need each reader a second
time.

## One node per package

| entry | nodes | output |
| --- | --- | --- |
| a package | a fixed-output fetch, `ms-2.1.3.tgz` | the tarball |
|  | a node, `ms-2.1.3` | the unpacked package |
| a package with an install script | the same two | the package after `preinstall`, `install` and `postinstall` |
| the project | `<name>` | the project, its tree and its programs |

**The planner writes the fetches.** The Python and Cargo lanes fetch on
the host. A fetch here is a `Fixed` node from the planner, with the hash
of the lock. Its path is known before it builds (`fixed_output_path` in
`pydd.core`), so the next node names the path itself, not a placeholder.

**A node holds no dependency.** Node resolves a module from its real
path. A package linked from the store looks for its dependencies beside
the store path, and finds none. So a node holds only its own package,
and the root copies each node to the path that the lock gives it. Node
then resolves every `require` as it would after `npm ci`, with nested
versions too. The cost is disk space at build time; the store's
deduplication shares the files.

The other layout gives each node links to its dependencies, as pnpm
does. npm's trees have cycles, and a derivation cannot depend on itself,
so that layout needs one node for each cycle. This layout needs none.

## Install scripts

A package with `hasInstallScript` runs `preinstall`, `install` and
`postinstall` in its own node. A package with `binding.gyp` and no such
script runs `node-gyp rebuild`, as npm does. The node first copies the
part of the tree that the script can `require`: every package that the
lock resolves from the package, transitively, with each program in
`node_modules/.bin`.

The script gets what npm's run-script gives it, without a network:

| variable | value |
| --- | --- |
| `PATH` | each `node_modules/.bin` above the package, `node-gyp`, Node, stdenv's tools, a C compiler, Python |
| `npm_config_nodedir` | Node of the menu, so node-gyp downloads no headers |
| `npm_config_build_from_source` | `true`, so prebuild-install and node-pre-gyp download no binary |
| `npm_package_name`, `npm_package_version`, `npm_lifecycle_event` | as npm sets them |

`tests/node/native-npm` builds better-sqlite3. Its script tries
`prebuild-install`, which refuses, and then node-gyp compiles the SQLite
that the package bundles.

Install scripts that depend on each other in a cycle are refused, with
the cycle in the message. npm runs them in an order of its own choice.

## Other sources

| `resolved` in the lock | source |
| --- | --- |
| `https://...` with `integrity` | a fixed-output fetch of the tarball |
| `git+ssh://git@github.com/...#<commit>`, GitLab, sourcehut | a fixed-output fetch of the commit's archive, unpacked |
| a path in the project, `file:...` | the project's own file |
| a workspace member, with `link: true` | a link in the root, to the member |

**A git dependency needs its hash.** The lock names the commit and no
hash. The call gives `outputHashes."<name>-<version>"`, the hash of the
unpacked archive, as `fetchFromGitHub` takes it. The planner's error
names the URL, and `nix store prefetch-file --unpack <url>` prints the
hash. nixpkgs' `prefetch-npm-deps` maps the URLs of the three hosts the
same way. A git dependency with a `prepare` script is refused: npm runs
it with the package's development dependencies, and the lock does not
name those.

**`overrides.<name>.src` replaces a package**, with a tarball or a
directory, wherever the lock installs it. Its fetch is not written. This
is how a patched or a prebuilt package goes in, as nixpkgs'
`importNpmLock` takes `packageSourceOverrides`. `tests/node/git-dep`
replaces `ms`, and fetches `isarray` from a commit on GitHub.

## Native inputs

nixpkgs has no table of npm packages and their inputs, as
`defaultCrateOverrides` is for crates. So the lane has no hints. Two
sources give a package its tools, and the later one wins:

| source | example |
| --- | --- |
| the project | `"pydd": { "buildTools": { "canvas": ["pkg-config", "cairo.dev", "cairo"] } }` in `package.json` |
| the call | `buildTools.canvas = [ pkgs.pkg-config pkgs.cairo.dev pkgs.cairo ];`, `overrides.<package>.env` |

A tool reaches only the install script of its package, the same way as in
the Rust lane (`tool_env` of `pydd.core`).

## The project

The root copies the project, places every node, and links each workspace
member where the lock names it. It runs each script of `scripts`
(`[ "build" ]`) with `npm run --if-present`, in the project and in each
workspace member. With a pnpm, bun or Yarn lock it runs them in the
project only, as those tools do. The project builds in the build directory, and not in
`$out/lib/node_modules`: TypeScript, and api-extractor over it, skip
every file under a `node_modules`, the project's own included.

The root then removes what `omit` leaves out, and the directories that
this leaves empty. It links each program of the project and of its
workspace members into `bin`.

A program's `#!/usr/bin/env node` becomes the path of Node, as stdenv's
`patchShebangs` does. The packages' programs get the same.

## The modes

Each project of `tests/node` gives one result in five modes:
`nix run --file . test-node`.

| mode | Nix | writes through |
| --- | --- | --- |
| `cli` | master, `builder-rpc-v0` | `nix derivation add` |
| `huggorm` | master, `builder-rpc-v0` | huggorm |
| `drv` | master, `builder-rpc-v0` | `pydd-drv` |
| `recursive` | 2.35, recursive-nix | `nix derivation add`, root to `$out` |
| `out` | 2.35, no socket | the host makes every node; the planner writes the root to `$out` |

## `out`: the host makes the graph

**`out` exists for npm, and not for Cargo.** On released Nix with no
socket, a planner writes one derivation. Every node that its root names
must exist before the planner runs, as an input of it. Only cargo, in a
build, knows Rust's units, so the host cannot make them. The npm lock
names every node, so the host can.

`lib/node/graph.nix` is the planner's resolution a second time, in Nix:
the platform, Node's walk, the closure of an install script, the `.bin`
links, and the spec of each node. It must give the `.drv` that the
planner gives, byte for byte. `OutWriter` refuses a root that names any
other, so a difference fails the build and does not pass unseen. Three
rules make the two agree:

- **JSON as `builtins.toJSON` writes it**: sorted keys, no spaces, no
  escaped Unicode. Every list in a spec is sorted.
- **A tool is an input derivation.** A string with a package's path
  carries that derivation in Nix. So the menu gives the planner `refs`,
  the derivation and output of each such path, and a node names the
  derivation and not the path.
- **A fetch is named by its URL**, and not by the first path that needs
  it, so the order of a walk cannot change a name.

The planner then only chooses the root. The graph is static, as
`importNpmLock` makes it, and the evaluation reads the lock:

| nodemon, 807 lock entries | evaluator CPU | function calls |
| --- | --- | --- |
| `cli`, the planner reads the lock | 1.01 s | 0.27 M |
| `out`, the host reads the lock | 1.31 s | 1.99 M |

The cost of `cli` does not grow with the lock. The cost of `out` does.
Whether a lock is the whole plan decides whether an ecosystem has `out`
at all.

A variable in `overrides.<name>.env` that names a store path differs
between the two in `out`: the menu gives the planner the text, without
the derivation.

## What is missing

- A git dependency outside GitHub, GitLab and sourcehut, and one with a
  `prepare` script. The planner and the node refuse them by name.
- `npm test` as `root.tests`.
- Yarn 2 and newer, whose checksums Nix cannot check.
- `out` for pnpm, bun and Yarn.