Dependency Generators

Tools that generate deps TOML files from native lock files.

Overview

Each language has a generator that:

1. Reads native lock files (go.sum, Cargo.lock, uv.lock)
2. Extracts dependency information
3. Prefetches packages to get Nix hashes
4. Outputs deps TOML for Nix cell building

Generator Structure

Input

Native lock file format (varies by language).

Output

TOML file with dependencies:

# go-deps.toml example
[deps]
[deps."github.com/pkg/errors"]
version = "v0.9.1"
hash = "sha256-xyz..."

[deps."golang.org/x/sys"]
version = "v0.15.0"
hash = "sha256-abc..."

Existing Generators

godeps-gen (Go)

Located at cmd/godeps-gen/.

godeps-gen --prefetch -o go-deps.toml

Reads: go.mod, go.sum

rustdeps-gen (Rust)

Located at cmd/rustdeps-gen/.

rustdeps-gen --cargo-lock Cargo.lock -o rust-deps.toml

Reads: Cargo.lock

pydeps-gen (Python)

Located at cmd/pydeps-gen/.

pydeps-gen --lock pylock.toml -o python-deps.toml

Reads: pylock.toml, uv.lock, or requirements.txt

Rust Dependency Handling

Rust crates can have build.rs scripts that run during compilation. Since we can't run arbitrary code in Nix's sandbox, build script outputs must be handled manually.

The Standard Flow

Cargo.lock → rust-deps.toml → rust-deps-cell.nix → .turnkey/rustdeps/

1. Cargo.lock defines exact versions and dependency graph
2. rust-deps.toml adds Nix hashes for each crate (generated by rustdeps-gen)
3. rust-deps-cell.nix fetches crates and generates rules.star files
4. gen-rust-buck.py parses each crate's Cargo.toml to generate its rules.star

What Works Automatically

• Dependencies: Resolved from [dependencies] in Cargo.toml
• Features: Unified across the dependency graph (like Cargo does)
• Crate renaming: package = "real-name" in dependencies
• Proc-macros: Detected from [lib] proc-macro = true
• Edition: Read from package.edition (defaults to 2015)
• Crate root: Detected from [lib] path or standard locations

What Requires Manual Handling

Diagnosing Problems

Symptom: Undefined cfg Flag

Error:

error[E0425]: cannot find value `fast_arithmetic` in this scope

Diagnosis: The crate's build.rs sets this via cargo:rustc-cfg=fast_arithmetic="64".

Solution:

rustcFlagsRegistry = {
  serde_json = ["--cfg" ''fast_arithmetic=\"64\"''];
};

Symptom: Missing Generated File

Error:

error[E0432]: unresolved import `crate::private`

Diagnosis: The crate expects a file in OUT_DIR that build.rs generates.

Solution:

buildScriptFixups = {
  serde = { patchVersion, vendorPath, ... }: ''
    mkdir -p "$out/${vendorPath}/out_dir"
    cat > "$out/${vendorPath}/out_dir/private.rs" << 'EOF'
#[doc(hidden)]
pub mod __private${patchVersion} {
    pub use crate::private::*;
}
EOF
  '';
};

Symptom: Linker Error for Native Symbols

Error:

error: linking with `cc` failed: exit status: 1
  = note: undefined reference to `ring_core_0_17_14__OPENSSL_cpuid_setup'

Diagnosis: The crate has C/assembly code that build.rs compiles.

Solution: Complex fixup that compiles the native code (see ring example in defaults).

The Registry System

Configuration lives in your flake.nix under turnkey.toolchains.buck2:

{
  turnkey.toolchains = {
    buck2 = {
      enable = true;
      rustDepsFile = ./rust-deps.toml;

      # Rustc flags for build scripts that emit cfg directives
      rustcFlagsRegistry = {
        my_crate = ["--cfg" "my_flag"];
        "my_crate@1.2.3" = ["--cfg" "version_specific_flag"];
      };

      # Build script fixups for generated files
      buildScriptFixups = {
        my_crate = { patchVersion, vendorPath, ... }: ''
          mkdir -p "$out/${vendorPath}/out_dir"
          echo "// generated" > "$out/${vendorPath}/out_dir/generated.rs"
        '';
      };

      # Feature overrides (in separate file)
      rustFeaturesFile = ./rust-features.toml;
    };
  };
}

Version-Aware Lookup

Both registries support version-specific keys:

rustcFlagsRegistry = {
  # Catch-all for any version
  rustix = ["--cfg" "libc" "--cfg" "linux_like" "--cfg" "linux_kernel"];

  # Specific version override (takes precedence)
  "rustix@0.38.0" = ["--cfg" "libc" "--cfg" "linux_like"];
};

Resolution order:

1. "crate@version" - Exact versioned key
2. "crate" - Catch-all for any version
3. Default registry (if exists)
4. Empty (no flags/fixup)

Fixup Function Context

Fixup functions receive context about the crate:

buildScriptFixups = {
  my_crate = { crateName, version, patchVersion, key, vendorPath }: ''
    # crateName: "my_crate"
    # version: "1.2.3"
    # patchVersion: "3" (last component)
    # key: "my_crate@1.2.3"
    # vendorPath: "vendor/my_crate@1.2.3"

    echo "Building fixup for ${crateName} version ${version}"
    mkdir -p "$out/${vendorPath}/out_dir"
  '';
};

Nix Interpolation vs Shell Escaping

In Nix multiline strings ('' ... ''):

buildScriptFixups = {
  my_crate = { patchVersion, vendorPath, ... }: ''
    # CORRECT: ${patchVersion} is Nix interpolation
    MY_VAR="${patchVersion}"

    # WRONG: ''${patchVersion} escapes the $ for shell
    # This becomes literal ${patchVersion}, which is undefined
    MY_VAR="''${patchVersion}"  # Results in empty string!

    # CORRECT: $out is a shell variable (set by Nix's runCommand)
    echo "Output: $out"
  '';
};

Rule: Use ${var} for Nix variables, $var for shell variables.

Default Registries

Turnkey includes defaults for known problematic crates:

Default rustcFlagsRegistry:

{
  serde_json = ["--cfg" ''fast_arithmetic=\"64\"''];
  rustix = ["--cfg" "libc" "--cfg" "linux_like" "--cfg" "linux_kernel"];
}

Default buildScriptFixups:

• serde_core - Generates out_dir/private.rs
• serde - Generates out_dir/private.rs
• ring - Compiles native crypto library (~440 lines of build commands)

User-provided values override defaults (same key) or extend them (new keys).

Best Practices

1. Check build.rs first - Read the crate's build.rs to understand what it does
2. Start simple - Try rustcFlagsRegistry before complex fixups
3. Version your fixups - Use versioned keys if build.rs changes between versions
4. Document complex fixups - Explain what the original build.rs does

Debugging Tips

Inspect Generated rules.star Files

cat .turnkey/rustdeps/vendor/serde_json@1.0.140/rules.star

Look for:

• rustc_flags - Should include cfg flags
• env - Should include OUT_DIR if fixup crate
• deps - Dependencies resolved correctly

Check Fixup Output

ls -la .turnkey/rustdeps/vendor/ring@0.17.14/out_dir/

Trace Feature Resolution

grep -A20 "rust_library" .turnkey/rustdeps/vendor/serde@*/rules.star | grep features

Creating a New Generator

1. Create CLI Tool

Create a CLI tool in cmd/newlang-gen/:

// cmd/newlang-gen/main.go
package main

import (
    "flag"
    "os"
    // ...
)

func main() {
    lockFile := flag.String("lock", "newlang.lock", "Path to lock file")
    output := flag.String("o", "", "Output file (default: stdout)")
    prefetch := flag.Bool("prefetch", true, "Fetch Nix hashes")
    flag.Parse()

    // 1. Parse lock file
    deps := parseLockFile(*lockFile)

    // 2. Prefetch packages if requested
    if *prefetch {
        prefetchHashes(deps)
    }

    // 3. Output TOML
    outputTOML(deps, *output)
}

2. Create Cell Builder

Create nix/buck2/newlang-deps-cell.nix:

{ pkgs, lib, depsFile }:

let
  deps = builtins.fromTOML (builtins.readFile depsFile);

  fetchDep = name: info:
    pkgs.fetchurl {
      url = info.url;
      hash = info.hash;
    };

  depSources = lib.mapAttrs fetchDep deps.deps;
in
pkgs.runCommand "newlang-deps-cell" {} ''
  mkdir -p $out

  # Generate cell .buckconfig
  cat > $out/.buckconfig << 'EOF'
  [cells]
      newlang-deps = .
  [buildfile]
      name = rules.star
  EOF

  # Generate rules.star for each dependency
  ${lib.concatStrings (lib.mapAttrsToList (name: src: ''
    mkdir -p $out/${name}
    cp -r ${src}/* $out/${name}/
    cat > $out/${name}/rules.star << 'EOF'
    # Generated build rules for ${name}
    newlang_library(
        name = "${lib.last (lib.splitString "/" name)}",
        srcs = glob(["*.newlang"]),
        visibility = ["PUBLIC"],
    )
    EOF
  '') depSources)}
''

3. Add to Devenv Module

Update nix/devenv/turnkey/buck2.nix to support the new language.

4. Add Configuration Options

Add options for deps file path and any language-specific configuration.

Testing

# Generate deps file
newlang-gen > newlang-deps.toml

# Verify it's valid TOML
nix eval --expr 'builtins.fromTOML (builtins.readFile ./newlang-deps.toml)'

# Build the cell
nix build .#newlang-deps-cell

# Check generated content
ls result/