Buck2 Cell Generation

This document describes how Turnkey generates Buck2 cells from Nix derivations.

Overview

Turnkey generates several types of Buck2 cells:

Toolchains cell - Language toolchains (Go, Rust, Python, etc.)
Dependency cells - Third-party packages (godeps, rustdeps, pydeps)
Prelude cell - Buck2 prelude with extensions

All cells are built as Nix derivations and symlinked into .turnkey/.

Toolchains Cell

Located at nix/buck2/toolchains-cell.nix. Generated from nix/buck2/mappings.nix.

nix/buck2/toolchains-cell-package.nix builds it for a shell's toolchain.toml. The flake-parts module builds one per shell, hands it to the shell (turnkey.buck2.toolchainsCell), which symlinks it at .turnkey/toolchains, and exposes the default shell's as the toolchains-cell package. The composition daemon builds that package like every other *-cell.

Mapping Structure

{
  go = {
    skip = false;
    targets = [{
      name = "go";
      rule = "system_go_toolchain";
      load = "@prelude//toolchains/go:system_go_toolchain.bzl";
      visibility = [ "PUBLIC" ];
      dynamicAttrs = registry: {
        go_binary = "${registry.go}/bin/go";
      };
    }];
    implicitDependencies = [ "python" "cxx" ];
    runtimeDependencies = [ ];
  };
}

Generated Output

rules.star is generated with:

Load statements for each rule
Rule instantiations with configured attributes
Visibility set to PUBLIC

Adding Toolchain Mappings

Edit nix/buck2/mappings.nix:

mylang = {
  skip = false;
  targets = [{
    name = "mylang";
    rule = "system_mylang_toolchain";
    load = "@prelude//mylang:toolchain.bzl";
    visibility = [ "PUBLIC" ];
    dynamicAttrs = registry: {
      compiler_path = "${registry.mylang}/bin/mylang";
    };
  }];
  implicitDependencies = [ ];
  runtimeDependencies = [ ];
};

Go Dependency Cell

Built by the Go adapter, nix/lib/deps-cell/adapters/go.nix. Like the Rust cell, it is a write-once cell (ADR 0004, ADR 0008): a real directory in the project, with one store link per Go module and one alias package per Go package.

Cell Structure

.turnkey/godeps/
├── .buckconfig                       # Cell identity
├── .deps-file-sha256                 # the go-deps.toml it was built from
├── _store/
│   └── <hash>-dep-go-golang-org-x-sys -> /nix/store/<hash>-dep-go-golang-org-x-sys
│       ├── cpu/rules.star            # go_library, generated in the module's derivation
│       └── unix/rules.star
└── vendor/
    └── golang.org/x/sys/
        ├── cpu/rules.star            # alias -> //_store/<hash>-dep-go-golang-org-x-sys/cpu:cpu
        └── unix/rules.star           # alias -> //_store/<hash>-dep-go-golang-org-x-sys/unix:unix

The vendor/ tree mirrors Go import paths, so labels name the import path and never a version. Alias packages nest where modules do (vendor/cloud.google.com/go and vendor/cloud.google.com/go/storage).

Process

tk sync runs godeps-gen, which records each module's version and hash in go-deps.toml, and the go.work members under [members].
Nix builds one package per module (mkGoDepPackage): the module's source from the Go proxy, its fixup, its user patches (.turnkey/patches/godeps/vendor/<module path>/), the assembly headers its .s files include, and its rules.star files, generated by buckgen from the module's own source and the cell-wide settings (cell name, platforms, allowed build tags, Go minor version). Its targets output lists the Go packages rendered (<subdir> <target>), its imports output the import paths they reference.
mkCellIndex builds the cell index: one package per Go package, at vendor/<import path>, with its module's store path and its subdir there. An import path two modules offer goes to the longer module path. A referenced import path a go.work member owns becomes a forwarding alias package, to the member's target in the root cell (root//<member dir>/<rest>:<last component>).
The shell runs tk materialize with the index, as for the Rust cell.

A module bump rebuilds that module's derivation alone, and buck2 recompiles its reverse dependencies alone. Adding or removing a go.work member rebuilds no module: only the index and its forwarding aliases change.

Cell Configuration

# .buckconfig
[cells]
    godeps = .
    prelude = bundled://

[buildfile]
    name = rules.star

Generated rules.star Files

Each Go package a configured platform builds gets a go_library target, in its module's store path:

# _store/<hash>-dep-go-github-com-spf13-cobra/rules.star
go_library(
    name = "cobra",
    package_name = "github.com/spf13/cobra",
    srcs = native.glob(["*.go", "*.s", "*.h", "*.c", "*.cc", "*.cpp", "*.S"]),
    header_namespace = "",
    visibility = ["PUBLIC"],
    deps = [
        "godeps//vendor/github.com/inconshreveable/mousetrap:mousetrap",
        "godeps//vendor/github.com/spf13/pflag:pflag",
    ],
)

Deps that only some platforms or build tags need are a select().

Target Path Format

When writing rules.star files that depend on packages from the godeps cell:

godeps//vendor/<import-path>:<target-name>

Where:

godeps// - the cell alias (configured in .buckconfig)
vendor/ - required prefix - all packages live under vendor/
<import-path> - the full Go import path
<target-name> - the directory name (last path component), NOT the package name

Examples:

Go Import	Correct Buck2 Target	Why
`github.com/spf13/cobra`	`godeps//vendor/github.com/spf13/cobra:cobra`	Target is `cobra` (dir name)
`github.com/pelletier/go-toml/v2`	`godeps//vendor/github.com/pelletier/go-toml/v2:v2`	Target is `v2` (dir name)
`golang.org/x/sys/unix`	`godeps//vendor/golang.org/x/sys/unix:unix`	Target is `unix` (dir name)

Common Mistakes:

# WRONG - missing vendor/ prefix
deps = ["godeps//github.com/spf13/cobra:cobra"]

# WRONG - using package name instead of directory name for versioned imports
deps = ["godeps//vendor/github.com/pelletier/go-toml/v2:go-toml"]

# CORRECT
deps = ["godeps//vendor/github.com/pelletier/go-toml/v2:v2"]

Import Path Resolution

The go_library rule's package_name makes the Go compiler see the import path, whatever the target's label:

Buck2 target: godeps//vendor/github.com/spf13/cobra:cobra
Go import: import "github.com/spf13/cobra"

Rust Dependency Cell

Built by the Rust adapter, nix/lib/deps-cell/adapters/rust.nix. Like the Go cell, it is a write-once cell (ADR 0004): a real directory in the project, not a symlink to one store path.

Process

tk sync runs rustdeps-gen, which records each crate's hash and its package slice in rust-deps.toml: features and resolved dependencies as cargo's feature resolver gives them, per platform (ADR 0006).
Nix builds one package per crate (mkRustCrates): the crate's source, its fixup, its user patches, and its rules.star, generated by rust-rules-gen from the crate's own slice and fixup alone. A second output lists the crate's target names.
mkCellIndex builds the cell index: each package's path, store path and targets, the unversioned aliases (highest version), the cell's .buckconfig, and the SHA-256 of rust-deps.toml. The flake exports it as rustdeps-index.
The shell runs tk materialize with the index: it keeps .turnkey/rustdeps in line, with a store link per package under _store/, never retargeted, and an alias package per versioned and unversioned name under vendor/, rewritten only when it changes.

Special Handling

Rust crates may require:

rustc flags - Build scripts that emit cargo:rustc-cfg
Generated files - Build scripts that generate .rs files
Native code - Build scripts that compile C/assembly

See Dependency Generators for handling these cases.

Python Dependency Cell

Built by the Python adapter, nix/lib/deps-cell/adapters/python.nix. Like the Rust and Go cells, it is a write-once cell (ADR 0004, ADR 0010): one store link per distribution, and one alias package per distribution.

Cell Structure

.turnkey/pydeps/
├── .buckconfig                       # Cell identity
├── .deps-file-sha256                 # the python-deps.toml it was built from
├── _store/
│   └── <hash>-dep-python-six-1.17.0 -> /nix/store/<hash>-dep-python-six-1.17.0
│       ├── six.py                    # the unpacked wheel: .py files are srcs,
│       ├── six-1.17.0.dist-info/     # every other file a resource
│       └── rules.star                # python_library, generated in the distribution's derivation
└── vendor/
    └── six/rules.star                # alias -> //_store/<hash>-dep-python-six-1.17.0:six

Labels are pydeps//vendor/<name>:<name>, with <name> the distribution's key in python-deps.toml. python-deps.toml holds one version per name (pydeps-gen fails on a forked lock), so there are no version alias packages.

Process

tk sync runs pydeps-gen, which records each distribution's version, and its pure (py3-none-any) wheel's URL and unpacked hash, from pylock.toml (python-deps.toml schema 3), and its dependencies, markers and extras from uv.lock. A distribution with no pure wheel fails it (ADR 0013).
Nix builds one package per distribution (mkPythonDepPackage): its wheel from PyPI, unpacked and installed (installWheel: <name>.data/'s purelib and platlib merged into the root, the rest dropped), its fixup, its user patches (.turnkey/patches/pydeps/vendor/<name>/), and its rules.star, written by pydeps-cell from the distribution's package slice (its dependencies and its requested extras', narrowed by sliceOf to the distributions the cell holds), the platforms' conditions and the Python toolchain's version. Its targets output holds its one target, <name>.
mkCellIndex builds the cell index: one package per distribution, at vendor/<name>, with its store path.
The shell runs tk materialize with the index, as for the Rust cell.

A version bump rebuilds that distribution's derivation, and those whose slice names it only if their slice changed; buck2 re-runs its reverse dependencies alone.

Solidity Dependency Cell

Built by the Solidity adapter, nix/lib/deps-cell/adapters/solidity.nix. Like the Go and Rust cells, it is a write-once cell (ADR 0004, ADR 0011), and the first whose users address its root package.

Cell Structure

.turnkey/soldeps/
├── .buckconfig
├── .deps-file-sha256                 # the solidity-deps.toml it was built from
├── rules.star                        # bundle and one alias per package, from the index's root
├── _store/
│   └── <hash>-dep-sol-forge_std-1.8.0 -> /nix/store/<hash>-dep-sol-forge_std-1.8.0
│       └── rules.star                # forge_std (.sol files) and forge_std_all filegroups
└── vendor/
    └── forge-std/
        ├── rules.star                # alias -> //_store/<hash>-dep-sol-forge_std-1.8.0:forge_std(_all)
        └── src -> /nix/store/<hash>-dep-sol-forge_std-1.8.0/src   # for native forge

Process

tk sync runs soldeps-gen, which writes one [[package]] per name to solidity-deps.toml (a name declared twice resolves to one package, or fails), and the root remappings.txt.
Nix builds one package per Solidity package (mkSolDepPackage): the npm tarball or git archive, its fixup, its user patches (.turnkey/patches/soldeps/vendor/<name>/), and its rules.star. Its targets output lists <target> and <target>_all.
mkCellIndex builds the cell index: one package per Solidity package at vendor/<name>, with no version aliases, and root, the root package's rules.star: bundle, which maps each vendor/<name> to //vendor/<name>:<target>_all, and an alias per package. Packages are exposed.
The shell runs tk materialize with the index. It writes root as it is, as it writes .buckconfig, and links each exposed package's store entries beside its alias package's rules.star. Native forge reads the cell in place through the root remappings.txt, so it needs the files at vendor/<name>/. Those links follow a bump to the new store path; buck2 never reads through them, since the alias package globs nothing.

A bump rebuilds the bumped package's derivation and the index. Every Solidity action stages the whole bundle, so they all re-run, and no action of another language does.

JavaScript Dependency Cell

Built by the JavaScript adapter, nix/lib/deps-cell/adapters/javascript.nix, as a write-once cell that separates a package's contents from where it sits in the graph (ADR 0012).

Cell Structure

.turnkey/jsdeps/
├── .buckconfig
├── .deps-file-sha256                 # the js-deps.toml it was built from
├── rules.star                        # the instance graph, from the index's root
├── _store/
│   └── <hash>-dep-js-micromatch-4.0.8 -> /nix/store/<hash>-dep-js-micromatch-4.0.8
│       └── rules.star                # files: a filegroup of the package
└── vendor/
    └── micromatch@4.0.8/
        └── rules.star                # alias files -> //_store/<hash>-dep-js-micromatch-4.0.8:files

The root rules.star loads @prelude//typescript:npm.bzl (nix/buck2/prelude-extensions/typescript/npm.bzl). The rules live in the prelude, not in the cell: a transitive set's type must be defined once.

npm_instance, one per [[instance]], named after pnpm's node_modules/.pnpm directory for its key (micromatch@4.0.8, react-dom@18.2.0_react@18.2.0), cut and hashed past 240 bytes. Its deps are by import name, and its optional deps that install on some platforms only are a select().
npm_component, one per dependency cycle (a strongly connected component of instances, computed at evaluation), and an npm_member forwarding to it per instance in the cycle.
alias, one per [direct] entry, named by the npm name (@types/micromatch). Instances are private to the cell's root package.

An instance's action copies its files with every link dereferenced into node_modules/<name>/, and links each dependency beside it, relative to its own output, into the dependency's. The output has no content-based path, and the links are passed with ignore_artifacts, so a link is keyed by its path: a dependency's content change re-runs only what copies it. NpmPackageInfo carries the package directory and the transitive set of instance outputs. typescript_library and typescript_binary link their npm_deps into a symlinked_dir node_modules and carry that set as hidden inputs (compile.bzl).

Process

tk sync runs jsdeps-gen, which writes [[package]] (one per name@version), [[instance]] (one per pnpm snapshot, with its dependencies resolved to instance keys) and [direct].
Nix builds one package per [[package]] (mkJsDepPackage): the npm tarball (or the project's own copy, buck2.javascript.tarballs), its fixup, its user patches (.turnkey/patches/jsdeps/vendor/<name>@<version>/), and its rules.star. Its targets output lists files.
mkCellIndex builds the cell index: one package per [[package]] at vendor/<name>@<version>, and root, the instance graph.
The shell runs tk materialize with the index.

A bump rebuilds the bumped package's derivation and the index, and re-runs the instances that link it, transitively, and their consumers.

Cell Configuration

Each cell gets a .buckconfig:

[cells]
    cellname = .
    prelude = path/to/prelude

[buildfile]
    name = rules.star

Dual-Build Compatibility

A key design goal is that code builds with both native tools and Buck2:

# Native build (uses go.mod directly)
go build ./...

# Buck2 build (uses generated cell)
buck2 build //...

This is achieved by:

No import rewriting - Go code uses standard import paths (github.com/foo/bar)
importpath attribute - Buck2's go_library rule's importpath tells the compiler the correct path
Nix-managed deps - Dependencies fetched by Nix, not vendored in repo

Adding New Dependency Cell Types

To add support for a new language:

Create deps generator (e.g., newlang-deps-gen)
Create cell builder (an adapter in nix/lib/deps-cell/adapters/)
Add the language's record to nix/buck2/languages.nix
Add configuration options to nix/buck2/options.nix

Debugging

Inspect Generated rules.star Files

cat .turnkey/godeps/vendor/github.com/spf13/cobra/rules.star          # the alias
cat .turnkey/godeps/_store/*-dep-go-github-com-spf13-cobra/rules.star  # the go_library

Check Cell Contents

ls -la .turnkey/godeps/vendor/

Verify Cell Configuration

cat .turnkey/godeps/.buckconfig

List Available Targets

buck2 targets godeps//...

Keyboard shortcuts

Turnkey Developer Manual