Dependency Generators

Tools that generate deps TOML files from native lock files.

Overview

Each language has a generator that:

1. Reads native lock files (go.sum, Cargo.lock, uv.lock)
2. Extracts dependency information
3. Prefetches packages to get Nix hashes
4. Outputs deps TOML for Nix cell building

Generator Structure

Input

Native lock file format (varies by language).

Output

TOML file with dependencies:

# go-deps.toml example
[deps]
[deps."github.com/pkg/errors"]
version = "v0.9.1"
hash = "sha256-xyz..."

[deps."golang.org/x/sys"]
version = "v0.15.0"
hash = "sha256-abc..."

What generators share

A generator's own code is its lock file parsing and its record type. The rest is the same for every generator:

• Flags: -o, --output; prefetching on by default, --no-prefetch to skip it; --no-cache to bypass turnkey's prefetch cache.
• Prefetching: the Nix SRI hash of a URL, of its unpacked contents for archives fetched with fetchzip, cached across runs and generators.
• Output: a header saying which generator wrote the file from what, and that tk sync regenerates it, then the record.

The Rust generators take all three from src/rust/deps-gen-kit: OutputArgs and PrefetchArgs to flatten into their clap Args, the Prefetcher seam (NixPrefetcher on the prefetch-cache crate, and MemoryPrefetcher for tests), and OutputArgs::write for a serde-serialized record. godeps-gen flattens the same flags, but prefetches through one nix-prefetch-cached --batch call (built on the same crate) and writes go-deps.toml line by line, in the layout its Go version wrote.

tk sync runs a generator from its language's sync rule (nix/buck2/languages.nix) and reads the deps file from stdout.

Existing Generators

godeps-gen (Go)

Located at src/cmd/godeps-gen/ (Rust).

godeps-gen -o go-deps.toml

Reads: go.work and its members' go.mod/go.sum, or go.mod, go.sum

rustdeps-gen (Rust)

Located at cmd/rustdeps-gen/.

rustdeps-gen --cargo-lock Cargo.lock \
  --platform linux-x86_64=x86_64-unknown-linux-gnu \
  --platform macos-arm64=aarch64-apple-darwin \
  -o rust-deps.toml

Reads: Cargo.lock and the workspace's manifests. Runs cargo tree once per --platform (<os>-<cpu>=<rust target triple>) and cargo metadata, both --locked, to record each crate's package slice: its features and normal dependencies as Cargo's feature resolver resolves them, each with the platforms it applies on (ADR 0006). It lists the workspace members' Cargo.toml files under manifests, which the Rust sync rule's target_sources makes sources of the file.

pydeps-gen (Python)

Located at cmd/pydeps-gen/.

pydeps-gen --lock pylock.toml -o python-deps.toml

Reads: pylock.toml, uv.lock, or requirements.txt

Records each distribution's pure (py3-none-any) wheel, hashed unpacked, and fails naming any distribution that has none (ADR 0013).

Go Dependency Handling

go.mod / go.work → go-deps.toml → one package per module + cell index (nix/lib/deps-cell) → tk materialize → .turnkey/godeps/

1. go.mod, or a root go.work and its members' go.mod files, define the build list; Go's own module resolution picks one version per module (ADR 0007)
2. go-deps.toml adds each module's Nix hash, and lists the go.work members (generated by godeps-gen)
3. Each module's own derivation (mkGoDepPackage in nix/lib/deps-cell/adapters/go.nix) applies the module's fixup and user patches, adds the assembly headers, and runs buckgen (src/cmd/buckgen/) on the module alone: one rules.star per Go package, labelled godeps//vendor/<import path>:<last component>. Two more outputs, targets and imports, list the Go packages rendered and the import paths they reference.
4. The cell index (godeps-index) lists each Go package's path, its module's store path and its subdirectory there, and a forwarding alias package for each referenced import path a go.work member owns (ADR 0008). tk materialize lays the cell out from it.

Rust Dependency Handling

Rust crates can have build.rs scripts that run during compilation. Since we can't run arbitrary code in Nix's sandbox, build script outputs must be handled manually.

The Standard Flow

Cargo.lock → rust-deps.toml → one package per crate + cell index (nix/lib/deps-cell) → tk materialize → .turnkey/rustdeps/

1. Cargo.lock defines exact versions and dependency graph
2. rust-deps.toml adds Nix hashes for each crate, and its package slice (generated by rustdeps-gen from cargo tree and cargo metadata)
3. Each crate's own derivation (mkRustCrates in nix/lib/deps-cell/adapters/rust.nix) applies the crate's fixup and user patches, and runs rust-rules-gen (src/cmd/rust-rules-gen/). That generates the crate's rules.star from the crate's directory, its slice, its own fixup, the platforms and the host, and nothing of any other crate: checks.rust-crate-isolation holds it to that. A second output, targets, lists the target names.
4. The cell index (rustdeps-index, built by mkCellIndex in nix/lib/deps-cell/default.nix) lists each package's path, store path and target names, the unversioned alias packages, the cell's .buckconfig and the SHA-256 of rust-deps.toml. tk materialize lays the cell out from it (ADR 0004).

rust-rules-gen is compiled because every crate's derivation starts it: on macOS, starting Python costs from about a second to about 40 s per process under parallel builds, which put a cold build of the per-crate derivations at roughly 15 minutes.

What Works Automatically

• Dependencies: As cargo resolves them, per platform (the package slice)
• Features: As cargo's feature resolver enables them, per platform
• Crate renaming: package = "real-name", and libraries named differently from their package
• Proc-macros: Detected from [lib] proc-macro = true
• Edition: Read from package.edition (defaults to 2015)
• Crate root: Detected from [lib] path or standard locations

What Requires a Fixup

Buck2 never runs a crate's build.rs. What a build script would produce comes from a fixup instead: a record for the crate in a fixup set, a module of class turnkeyFixups (ADR 0003). A repository brings fixup sets through turnkey.toolchains.buck2.fixups; the user manual's Dependency Fixups page covers bringing and publishing them. This section covers writing one and how turnkey applies it.

Every locked crate that has a build script needs a fixup whose buildScript either generates its output or says skip = true, even when its rustcFlags stand in for everything it does. Otherwise the cell fails to build, naming the crate. When one of turnkey's published families accounts for it, the error names the module to import.

Diagnosing Problems

Symptom: A crate has a build.rs, and no fixup says what stands in for it

The cell build fails with:

error: turnkey: zerocopy 0.8.37 has a build.rs, and no fixup says what stands in for it; give it one in turnkey.toolchains.buck2.fixups: ...

Diagnosis: read the crate's build.rs. If it only probes the rustc version or the target, or emits cfgs for features you don't use, the build needs nothing from it: rust.zerocopy.buildScript.skip = true. Otherwise, give it the fixup the symptoms below describe.

Symptom: Undefined cfg Flag

Error:

error[E0425]: cannot find value `fast_arithmetic` in this scope

Diagnosis: The crate's build.rs sets this via cargo:rustc-cfg=fast_arithmetic="64".

Solution:

rust.serde_json = {
  buildScript.skip = true; # the flag is all it produces
  rustcFlags = [ "--cfg" ''fast_arithmetic="64"'' ];
};

Symptom: Missing Generated File

Error:

error[E0432]: unresolved import `crate::private`

Diagnosis: The crate expects a file in OUT_DIR that build.rs generates.

Solution:

rust.serde.buildScript.generate = ctx: ''
  cat > "$OUT_DIR/private.rs" << 'EOF'
  #[doc(hidden)]
  pub mod __private${ctx.versionParts.patch} {
      pub use crate::private::*;
  }
  EOF
'';

A crate whose fixup generates output gets OUT_DIR = "out_dir" in its rules.

Symptom: Linker Error for Native Symbols

Error:

error: linking with `cc` failed: exit status: 1
  = note: undefined reference to `ring_core_0_17_14__OPENSSL_cpuid_setup'

Diagnosis: The crate has C/assembly code that build.rs compiles.

Solution: a build script that compiles it into $OUT_DIR, and the library it produces in nativeLibraries. See nix/fixups/rust/ring.nix and nix/fixups/rust/tree-sitter.nix.

The Fixup Record

A Rust fixup, as nix/lib/fixups/schema.nix declares it:

rust.my_crate = {
  # What stands in for build.rs: exactly one of generate or skip
  buildScript.generate = ctx: "...";   # or a string; or: buildScript.skip = true;

  rustcFlags = [ "--cfg" "my_flag" ];  # every platform
  env.MY_VAR = "value";                # the crate's compile environment
  patches = [ ./fix.patch ];           # -p1, relative to the crate's root
  nativeLibraries = [
    {
      name = ctx: "my_lib_${ctx.versionParts.patch}"; # a string, or a function of the context
      staticLib = "out_dir/libmy_lib.a";              # relative to the crate
      linkSearchPath = "out_dir";                     # the default
    }
  ];

  # Overlays: declarative additions per OS, per CPU, or per OS and CPU
  # pair, as select()s in the rules. A platform gets list fields from its
  # OS's overlay, then its CPU's, then its pair's. Two overlays giving one
  # platform an env variable different values fail evaluation. No build
  # script here: a crate has one, branching on ctx.platform.
  os.linux.rustcFlags = [ "--cfg" "linux_like" ];
  cpu.arm64.env.MY_ARCH = "arm64";
  platform."macos-arm64".rustcFlags = [ "--cfg" "apple_silicon" ];

  # Fields for the locked versions whose bounds hold; every match applies
  versions = [
    {
      when = { atLeast = "0.17"; below = "0.18"; };
      rustcFlags = [ "--cfg" "v017" ];
    }
  ];

  enable = true; # false drops a fixup an imported set brings
};

Every language's fixups have enable, patches, env and versions, keyed by the dependency's name in its own ecosystem (go."github.com/foo/bar", python.requests, …). Patches apply in every language; env is Rust-only for now, and an error elsewhere.

Merging is the module system's: lists concatenate in import order, and two sets giving one crate different build scripts, or one env variable different values, fail evaluation naming both files. Resolve it with lib.mkForce, disabledModules, or enable = false.

The Fixup Context

buildScript.generate and native library fields that are functions receive:

ctx = {
  name = "my_crate";
  version = "1.2.3-rc.1";
  versionParts = { major = "1"; minor = "2"; patch = "3"; pre = "rc.1"; };
  platform = { system = "aarch64-darwin"; os = "macos"; cpu = "arm64"; };
  pkgs = <nixpkgs>;
  lib = <nixpkgs lib>;
};

platform is the platform the cell is built on, which is the only one a native library the build script compiles exists for. turnkey links nativeLibraries on that platform only: building the crate for another platform fails in Buck2 ("no condition matched") rather than linking a missing library.

The build script runs in the crate's derivation, after its patches, with $CRATE_SRC its root and $OUT_DIR (created) its build script output directory.

Nix Interpolation vs Shell Escaping

In Nix multiline strings ('' ... ''):

rust.my_crate.buildScript.generate = ctx: ''
  # CORRECT: ${ctx.versionParts.patch} is Nix interpolation
  MY_VAR="${ctx.versionParts.patch}"

  # WRONG: ''${ctx.versionParts.patch} escapes the $ for the shell,
  # where it is undefined
  MY_VAR="''${ctx.versionParts.patch}"

  # CORRECT: $OUT_DIR is a shell variable
  echo "Output: $OUT_DIR"
'';

Rule: Use ${var} for Nix values, $var for shell variables.

turnkey's Own Fixups

turnkey's repository brings its own set, nix/fixups, one module per family: build-script-skips, fuser, nix, ring, rustix, serde, thiserror and tree-sitter. The flake publishes them as modules.turnkeyFixups.<family>, plus default importing them all. turnkey imports default itself; no other repository gets them unless it imports them.

Best Practices

1. Check build.rs first - Read the crate's build.rs to understand what it does
2. Start simple - skip, then rustcFlags, before a generating build script
3. Bound what is version-specific - Put it in a versions entry, so a new version is unaccounted for rather than silently wrong
4. Document complex fixups - Explain what the original build.rs does

Debugging Tips

Inspect Generated rules.star Files

cat .turnkey/rustdeps/vendor/serde_json@1.0.140/rules.star

Look for:

• rustc_flags - Should include cfg flags
• env - Should include OUT_DIR if the fixup generates build script output
• deps - Dependencies resolved correctly

Check Fixup Output

ls -la .turnkey/rustdeps/vendor/ring@0.17.14/out_dir/

Trace Feature Resolution

grep -A20 "rust_library" .turnkey/rustdeps/vendor/serde@*/rules.star | grep features

Creating a New Generator

1. Create CLI Tool

Create a CLI tool in cmd/newlang-gen/:

// cmd/newlang-gen/main.go
package main

import (
    "flag"
    "os"
    // ...
)

func main() {
    lockFile := flag.String("lock", "newlang.lock", "Path to lock file")
    output := flag.String("o", "", "Output file (default: stdout)")
    prefetch := flag.Bool("prefetch", true, "Fetch Nix hashes")
    flag.Parse()

    // 1. Parse lock file
    deps := parseLockFile(*lockFile)

    // 2. Prefetch packages if requested
    if *prefetch {
        prefetchHashes(deps)
    }

    // 3. Output TOML
    outputTOML(deps, *output)
}

2. Create Cell Builder

Add an adapter under nix/lib/deps-cell/adapters/, for example newlang.nix. A minimal cell builder looks like this:

{ pkgs, lib, depsFile }:

let
  deps = builtins.fromTOML (builtins.readFile depsFile);

  fetchDep = name: info:
    pkgs.fetchurl {
      url = info.url;
      hash = info.hash;
    };

  depSources = lib.mapAttrs fetchDep deps.deps;
in
pkgs.runCommand "newlang-deps-cell" {} ''
  mkdir -p $out

  # Generate cell .buckconfig
  cat > $out/.buckconfig << 'EOF'
  [cells]
      newlang-deps = .
  [buildfile]
      name = rules.star
  EOF

  # Generate rules.star for each dependency
  ${lib.concatStrings (lib.mapAttrsToList (name: src: ''
    mkdir -p $out/${name}
    cp -r ${src}/* $out/${name}/
    cat > $out/${name}/rules.star << 'EOF'
    # Generated build rules for ${name}
    newlang_library(
        name = "${lib.last (lib.splitString "/" name)}",
        srcs = glob(["*.newlang"]),
        visibility = ["PUBLIC"],
    )
    EOF
  '') depSources)}
''

3. Add the Language's Record

Add a record to nix/buck2/languages.nix: the cell name, depsFile (the deps file's default name), the generator package, mkCell (which calls the adapter) and syncRules. Everything else follows from the record: the flake-parts module builds the cell, and the devenv module adds the cell to .buckconfig, symlinks it under .turnkey/, puts the generator on the shell's PATH and writes the sync rules into .turnkey/sync.toml, with the cell and deps file in its [[languages]].

Rules sync (src/rust/rules-syncer, a library tk calls) reads [[languages]] and creates, for each language, the plug-in registered under the record's name in Mapper::new (src/rust/rules-syncer/src/mapper/mod.rs): a language without one is an error, so add the plug-in with the record, and update the checked-in src/rust/rules-syncer/testdata/sync.toml (checks.sync-config-contract prints the file to copy).

4. Add Configuration Options

Add the language's options to nix/buck2/options.nix: at least enable, cell and depsFile, plus whatever its sync rules read.

Testing

# Generate deps file
newlang-gen > newlang-deps.toml

# Verify it's valid TOML
nix eval --expr 'builtins.fromTOML (builtins.readFile ./newlang-deps.toml)'

# Build the cell
nix build .#newlang-deps-cell

# Check generated content
ls result/