Introduction

This manual is for developers who want to understand, extend, or contribute to Turnkey.

What You'll Learn

• Turnkey's architecture and design principles
• How the Nix module system works
• Adding new toolchains and language support
• Creating custom Buck2 rules
• Contributing to the project

Prerequisites

You should be familiar with:

• Nix - Flakes, derivations, module system
• Buck2 - Targets, rules, toolchains
• Starlark - Buck2's configuration language

Project Philosophy

Turnkey follows these principles:

1. Simplicity over features - Solve common cases elegantly
2. Declarative configuration - TOML in, working environment out
3. Reproducibility - Same inputs = same outputs, always
4. Composition - Build complex systems from simple parts
5. Transparency - Generated code should be readable

Repository Structure

turnkey/
├── nix/
│   ├── flake-parts/turnkey/  # Flake-parts integration
│   ├── devenv/turnkey/       # Devenv shell configuration
│   ├── registry/             # Default toolchain registry
│   ├── buck2/                # Buck2 cell generation
│   │   ├── prelude.nix       # Prelude derivation
│   │   ├── mappings.nix      # Toolchain mappings
│   │   └── prelude-extensions/
│   ├── packages/             # Tool packages
│   └── patches/              # Upstream patches
├── cmd/                      # CLI tools (Go)
├── docs/                     # Documentation
└── examples/                 # Example projects

Architecture Overview

Turnkey uses a layered architecture to transform simple TOML declarations into working development environments.

Data Flow

toolchain.toml
    ↓
Flake-parts module (perSystem options)
    ↓
Devenv module (shell configuration)
    ↓
Registry (name → package resolution)
    ↓
Buck2 cell generation (toolchains, deps)
    ↓
Working development shell

Layer Responsibilities

toolchain.toml

Simple declaration of needed tools:

[toolchains]
go = {}
rust = {}

Flake-Parts Module

• Exposes turnkey.toolchains options at perSystem level
• Builds dependency cells from deps files
• Creates devenv shell configurations
• Located at nix/flake-parts/turnkey/default.nix

Devenv Module

• Receives registry and declaration file
• Resolves toolchain names to packages
• Adds packages to shell PATH
• Generates Buck2 cells on shell entry
• Located at nix/devenv/turnkey/default.nix

Registry

• Maps toolchain names to Nix packages
• Versioned format: { go = { versions = { ... }; default = "..."; }; }
• Extensible by users via registryExtensions or custom overlays
• Default registry provided by teller

Buck2 Cells

• Generated at shell entry time
• Toolchains cell with language-specific rules
• Dependency cells (godeps, rustdeps, pydeps)
• Symlinked into .turnkey/

Key Design Decisions

1. Nix for package resolution - Leverages nixpkgs for reproducibility
2. Devenv for shell management - Proven shell environment tooling
3. Generated Buck2 cells - Dynamic, not committed to repo
4. Prelude from Nix - Pinned, patched, extended prelude

Module System

Turnkey uses NixOS-style modules for configuration.

Module Layers

Flake-Parts Module

Located at nix/flake-parts/turnkey/default.nix.

Provides perSystem options:

options.perSystem = mkPerSystemOption ({...}: {
  options.turnkey.toolchains = {
    enable = mkOption { type = types.bool; default = true; };
    declarationFiles = mkOption { type = types.attrsOf types.path; };
    registry = mkOption { type = types.lazyAttrsOf types.package; };
    buck2 = {
      enable = mkOption { ... };
      go.depsFile = mkOption { ... };
      rust.depsFile = mkOption { ... };
      # ...
    };
  };
});

Devenv Module

Located at nix/devenv/turnkey/default.nix.

Receives configuration from flake-parts:

options.turnkey = {
  enable = mkOption { type = types.bool; };
  declarationFile = mkOption { type = types.path; };
  registry = mkOption { type = types.lazyAttrsOf types.package; };
};

config = lib.mkIf cfg.enable {
  packages = resolvedPackages;
};

Configuration Flow

1. User sets turnkey.toolchains in their flake
2. Flake-parts module creates shell configs
3. Each shell config imports devenv module
4. Devenv module resolves packages from registry

Extending Options

Add new options in the appropriate module:

# In flake-parts module for user-facing API
options.turnkey.toolchains.myFeature = mkOption {
  type = types.bool;
  default = false;
  description = "Enable my feature";
};

# Pass to devenv module in mkShellConfig
turnkey.myFeature = cfg.myFeature;

Registry Pattern

The registry maps toolchain names to versioned Nix packages. The core registry library and default registry live in teller, a standalone Nix flake that turnkey depends on.

Structure

The default registry is provided by teller (registry/default.nix in the teller repo):

{ pkgs, lib ? pkgs.lib }:

let
  # Helper for single-version entries
  single = pkg: {
    versions = { "default" = pkg; };
    default = "default";
  };
in {
  go = single pkgs.go;
  rust = single pkgs.rustc;
  python = single pkgs.python3;
  # ...
}

Versioned Format

Each registry entry has:

<toolchain-name> = {
  versions = {
    "<version-string>" = <derivation>;
    # ...
  };
  default = "<version-string>";  # Must match a key in versions
};

For example, a multi-version entry:

go = {
  versions = {
    "1.21" = pkgs.go_1_21;
    "1.22" = pkgs.go_1_22;
    "1.23" = pkgs.go_1_23;
  };
  default = "1.23";
};

Design Principles

1. Versioned - Each toolchain can have multiple versions
2. Lazy evaluation - Only builds what's used
3. Composable - Multiple registries can be merged via overlays
4. User-overridable - Versions and defaults can be customized

Library Functions

Teller provides helpers in teller.lib (system-independent):

resolveTool

Resolves a toolchain from the registry:

# Usage
go = teller.lib.resolveTool registry "go" {};           # Use default
go122 = teller.lib.resolveTool registry "go" { version = "1.22"; };

resolveToolchains

Resolves all toolchains from a parsed toolchain.toml:

declaration = builtins.fromTOML (builtins.readFile ./toolchain.toml);
packages = teller.lib.resolveToolchains registry declaration;

mkRegistryOverlay

Creates overlays with two-level merging for registry composition:

overlays.default = teller.lib.mkRegistryOverlay (final: prev: {
  go = {
    versions = { "1.24" = final.go_1_24; };
    default = "1.24";
  };
});

When composed, versions are merged additively and default is overridden.

mkMetaPackage

Bundles multiple tools into a single derivation:

rust = {
  versions = {
    "1.80" = teller.lib.mkMetaPackage {
      inherit pkgs;
      name = "rust-1.80";
      components = {
        rustc = final.rustc;
        cargo = final.cargo;
        clippy = final.clippy;
        rustfmt = final.rustfmt;
      };
    };
  };
  default = "1.80";
};

Default Sourcing

The turnkey.toolchains flake-parts module options tellerLib and tellerRegistry default to the teller + toolbox setup that turnkey bundles, so consumers don't have to wire them up unless they need a private registry or an alternate teller revision.

The defaults are also exposed as standalone helpers on turnkey's lib output, reachable from any downstream flake:

# inputs.turnkey.lib.defaultTellerLib       : teller.lib (system-agnostic)
# inputs.turnkey.lib.defaultTellerRegistry  : system → registry attrset

So a typical consumer flake reduces to:

turnkey.toolchains = {
  enable = true;
  declarationFiles.default = ./toolchain.toml;
  buck2 = { ... };
};

Override scenarios:

• Private registry overlay — extend the toolbox-backed default with extra overlays:

  turnkey.toolchains.tellerRegistry =
    (import inputs.nixpkgs {
      inherit system;
      overlays = [
        inputs.teller.overlays.default
        inputs.toolbox.overlays.default
        inputs.my-org-overlay.overlays.default
      ];
    }).turnkeyRegistry;
  

• Alternate teller revision — pin a fork or a specific teller commit:

  turnkey.toolchains.tellerLib = inputs.my-teller-fork.lib;
  

Or reach the defaults directly for ad-hoc composition outside the module:

let
  registry = inputs.turnkey.lib.defaultTellerRegistry system;
  resolvedBuck2 = inputs.turnkey.lib.defaultTellerLib.resolveTool registry "buck2" {};
in ...

How It's Used

The flake-parts module injects tellerLib into the devenv module:

# In the devenv module:
turnkeyLib = cfg.tellerLib;

# Parse toolchain.toml and resolve all toolchains
declaration = builtins.fromTOML (builtins.readFile cfg.declarationFile);
packages = turnkeyLib.resolveToolchains cfg.registry declaration;

Extending the Registry

Via registryExtensions

In your flake.nix:

turnkey.toolchains = {
  registryExtensions = let
    single = pkg: { versions = { "default" = pkg; }; default = "default"; };
  in {
    mytool = single myCustomPackage;
    # Add versions to existing toolchain
    go = {
      versions = { "1.24" = pkgs.go_1_24; };
      default = "1.24";  # Override default
    };
  };
};

Via Custom Registry Overlay

For reusable registries, create a flake that exports an overlay:

# my-registry/flake.nix
{
  inputs.teller.url = "github:firefly-engineering/teller";

  outputs = { teller, ... }: {
    overlays.default = teller.lib.mkRegistryOverlay (final: prev: {
      zig = {
        versions = {
          "0.11" = final.zig_0_11;
          "0.12" = final.zig_0_12;
        };
        default = "0.12";
      };
    });
  };
}

Consumers compose overlays:

pkgs = import nixpkgs {
  overlays = [
    official-registry.overlays.default
    my-registry.overlays.default  # Versions merge!
  ];
};

Internal Tools

Dependency generators (godeps-gen, rustdeps-gen, etc.) are not in the registry. They're internal turnkey tools that are automatically included when the corresponding language is enabled.

Adding to Default Registry

To add a new standard toolchain, contribute to teller's registry/default.nix.

To add a turnkey-specific tool, add it to registryExtensions in turnkey's flake.nix.

# Single version (most common)
zig = single pkgs.zig;

# Multiple versions
nodejs = {
  versions = {
    "18" = pkgs.nodejs_18;
    "20" = pkgs.nodejs_20;
    "22" = pkgs.nodejs_22;
  };
  default = "20";
};

Buck2 Integration

This document describes how Turnkey integrates with Buck2 and covers the architecture of cell generation.

Overview

Turnkey generates Buck2 cells at shell entry time. Cells are directories that Buck2 treats as separate projects with their own configuration and build rules.

Cell Resolution

Buck2 cell resolution is entirely configuration-driven through .buckconfig files. There is no environment variable like CELL_PATH for overriding cell locations at runtime.

Primary Configuration: [cells] Section

Cells are defined in .buckconfig files:

[cells]
    root = .
    prelude = path/to/prelude
    toolchains = path/to/toolchains

Key points:

• Paths are relative to the directory containing the .buckconfig file
• Left side: cell alias (alphanumeric + underscores only)
• Right side: filesystem path

Configuration File Precedence

Buck2 reads configuration from multiple sources (highest to lowest precedence):

1. Command-line: --config, --config-file, --flagfile
2. .buckconfig.local (repo root)
3. .buckconfig (repo root)
4. .buckconfig.d/ folder (repo root)
5. ~/.buckconfig.local (user home)
6. ~/.buckconfig.d/ (user home)
7. /etc/buckconfig (global)
8. /etc/buckconfig.d/ (global)

Reference: app/buck2_common/src/legacy_configs/path.rs:35-60

Cell Override Restrictions

Buck2 explicitly bans overriding cell definitions using the --config command-line flag:

#![allow(unused)]
fn main() {
// app/buck2_common/src/legacy_configs/parser.rs:133-144
for banned_section in ["repositories", "cells"] {
    if config_pair.section == banned_section {
        return Err(
            ConfigArgumentParseError::CellOverrideViaCliConfig(banned_section).into(),
        );
    };
}
}

This means:

• buck2 --config cells.prelude=/path → ERROR
• buck2 --config repositories.toolchains=/path → ERROR

Solution: Use --config-file instead, which has no restrictions on [cells] sections.

Generated Cells

Toolchains Cell (.turnkey/toolchains/)

Contains toolchain rules for each declared language:

# Generated rules.star
load("@prelude//toolchains/go:system_go_toolchain.bzl", "system_go_toolchain")

system_go_toolchain(
    name = "go",
    visibility = ["PUBLIC"],
)

Prelude Cell (.turnkey/prelude/)

Symlink to Nix-built prelude with:

• Upstream buck2-prelude of the pinned buck2 release (nix/buck2/buck2-source.nix)
• Applied patches
• Custom extensions (TypeScript, mdbook, etc.)

Dependency Cells

• godeps/ - Go third-party packages
• rustdeps/ - Rust crates
• pydeps/ - Python packages

Toolchain Mappings

Located at nix/buck2/mappings.nix:

{
  go = {
    skip = false;
    targets = [{
      name = "go";
      rule = "system_go_toolchain";
      load = "@prelude//toolchains/go:system_go_toolchain.bzl";
      visibility = [ "PUBLIC" ];
      dynamicAttrs = registry: {
        go_binary = "${registry.go}/bin/go";
      };
    }];
    implicitDependencies = [ "python" "cxx" ];
    runtimeDependencies = [ ];
  };
}

Mapping Structure

Generation Process

1. Devenv shell entry hook runs
2. nix/devenv/turnkey/buck2.nix generates toolchains cell
3. Dependency cells built from deps files
4. Symlinks created in .turnkey/

Nix Integration Strategy

Turnkey uses symlinks to Nix store paths for Buck2 cells:

.turnkey/
├── toolchains -> /nix/store/...-turnkey-toolchains-cell
├── godeps     -> /nix/store/...-go-deps-cell
├── rustdeps   -> /nix/store/...-rust-deps-cell
├── pydeps     -> /nix/store/...-python-deps-cell
└── prelude    -> /nix/store/...-turnkey-prelude

Config File Solution

Since --config can't override cells but --config-file can, Turnkey generates a complete .buckconfig in the Nix store and symlinks to it:

# Generated .buckconfig in Nix store
[cells]
    root = .
    prelude = /nix/store/xxx-prelude
    toolchains = /nix/store/yyy-toolchains
    godeps = /nix/store/zzz-godeps

This allows each Nix environment to provide its own prelude/toolchains while sharing the same Buck2 project configuration.

Benefits

• Multiple shells from same git checkout
• Each shell can point to different toolchains/prelude
• No modification of version-controlled files
• Clean integration with Nix's environment management

Config File Capabilities

Value Interpolation

Reference other config values:

[custom]
    base_path = /nix/store/abc123

[cells]
    prelude = $(config custom.base_path)/prelude
    toolchains = $(config custom.base_path)/toolchains

Reference: app/buck2_common/src/legacy_configs/parser/resolver.rs:150-153

File Includes

Include other configuration files:

# Required include
<file:path/to/other.buckconfig>

# Optional include (no error if missing)
<?file:path/to/optional.buckconfig>

Limitations

No Environment Variable Substitution: Buck2 does not support $(env VAR_NAME) syntax in config files. Only $(config section.key) is supported.

This is why the generated config file approach is necessary.

External Cells

Buck2 supports external cells (bundled or git-based):

Bundled External Cells

[cells]
    prelude = prelude/

[external_cells]
    prelude = bundled

Git External Cells

[cells]
    prelude = prelude/

[external_cells]
    prelude = git

[external_cell_prelude]
    git_origin = https://github.com/facebook/buck2-prelude.git
    commit_hash = <40-char-sha1-hash>

Note: External cells don't solve the dynamic path problem since they still require configuration file changes.

Prelude Customization

The prelude is built by nix/buck2/prelude.nix:

1. Take the upstream buck2-prelude of the pinned buck2 release, from toolbox
2. Apply turnkey's patch set, nix/patches/prelude/*.patch
3. Copy extensions from nix/buck2/prelude-extensions/

See Prelude Extensions for adding custom rules.

Key Source Files (Buck2)

Key Source Files (Turnkey)

Build System Abstraction

Turnkey supports multiple build systems through abstraction layers that separate build-system-agnostic specifications from build-system-specific rule generation.

Overview

The abstraction pattern allows:

1. Single source of truth - Dependency specifications remain build-system-agnostic
2. Pluggable generators - Each build system implements its own rule generation
3. Easy extensibility - Adding new build systems requires only implementing generator protocols

┌─────────────────────────────────────────────────────────────────┐
│                   Generic Specification                         │
│  (NativeLibrarySpec, CellInfo, etc.)                            │
└─────────────────────────────────────────────────────────────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
     ┌────────────┐   ┌────────────┐   ┌────────────┐
     │   Buck2    │   │   Bazel    │   │   Future   │
     │ Generator  │   │ Generator  │   │ Generators │
     └────────────┘   └────────────┘   └────────────┘
              │               │               │
              ▼               ▼               ▼
     ┌────────────┐   ┌────────────┐   ┌────────────┐
     │ rules.star │   │ BUILD.bazel│   │    ...     │
     │ .buckconfig│   │ WORKSPACE  │   │            │
     └────────────┘   └────────────┘   └────────────┘

Native Library Generation

The Problem

Pre-compiled native libraries (like ring's crypto code) need different rules for each build system:

The Abstraction

NativeLibrarySpec

A build-system-agnostic specification for native libraries:

@dataclass
class NativeLibrarySpec:
    """Build-system-agnostic specification for a native library."""

    lib_name: str           # Target name (e.g., "ring_core_0_17_14__")
    static_lib_path: str    # Path to .a file (e.g., "out_dir/libring_core.a")
    link_search_path: str   # Rustc -L path (default: "out_dir")

This contains only the information needed to describe the library, not how to build it.

NativeLibraryGenerator Protocol

Build systems implement this protocol:

class NativeLibraryGenerator(Protocol):
    """Protocol for generating native library rules."""

    def generate(self, spec: NativeLibrarySpec) -> GeneratedRules:
        """Generate build rules for a native library."""
        ...

    @property
    def name(self) -> str:
        """The build system name (e.g., 'buck2', 'bazel')."""
        ...

GeneratedRules

The output from a generator:

@dataclass
class GeneratedRules:
    """Result of generating native library rules."""

    rules_content: str          # Generated rule definitions
    rules_to_load: list[str]    # Rules to load (e.g., ["prebuilt_cxx_library"])
    extra_deps: list[str]       # Dependencies to add to the crate
    extra_rustc_flags: list[str] # Rustc flags for linking

Implementation Examples

Buck2 Generator

class Buck2NativeLibraryGenerator:
    @property
    def name(self) -> str:
        return "buck2"

    def generate(self, spec: NativeLibrarySpec) -> GeneratedRules:
        lines = [
            "export_file(",
            f'    name = "{spec.lib_name}_file",',
            f'    src = "{spec.static_lib_path}",',
            '    visibility = ["PUBLIC"],',
            ")",
            "",
            "prebuilt_cxx_library(",
            f'    name = "{spec.lib_name}",',
            f'    static_lib = ":{spec.lib_name}_file",',
            "    link_whole = True,",
            '    preferred_linkage = "static",',
            '    visibility = ["PUBLIC"],',
            ")",
        ]

        return GeneratedRules(
            rules_content="\n".join(lines),
            rules_to_load=["prebuilt_cxx_library", "export_file"],
            extra_deps=[f":{spec.lib_name}"],
            extra_rustc_flags=[f"-Lnative={spec.link_search_path}"],
        )

Generated output:

export_file(
    name = "ring_core_0_17_14___file",
    src = "out_dir/libring_core_0_17_14__.a",
    visibility = ["PUBLIC"],
)

prebuilt_cxx_library(
    name = "ring_core_0_17_14__",
    static_lib = ":ring_core_0_17_14___file",
    link_whole = True,
    preferred_linkage = "static",
    visibility = ["PUBLIC"],
)

Bazel Generator

class BazelNativeLibraryGenerator:
    @property
    def name(self) -> str:
        return "bazel"

    def generate(self, spec: NativeLibrarySpec) -> GeneratedRules:
        lines = [
            "cc_import(",
            f'    name = "{spec.lib_name}",',
            f'    static_library = "{spec.static_lib_path}",',
            '    visibility = ["//visibility:public"],',
            ")",
        ]

        return GeneratedRules(
            rules_content="\n".join(lines),
            rules_to_load=["cc_import"],
            extra_deps=[f":{spec.lib_name}"],
            extra_rustc_flags=[f"-Lnative={spec.link_search_path}"],
        )

Generated output:

cc_import(
    name = "ring_core_0_17_14__",
    static_library = "out_dir/libring_core_0_17_14__.a",
    visibility = ["//visibility:public"],
)

File Locations

Usage in Generator

The generator.py uses the abstraction:

from python.buildsystem.native_library import NativeLibrarySpec
from python.buildsystem.buck2 import buck2_generator

def generate_buck_file(..., native_lib_info: dict | None = None) -> str:
    if native_lib_info:
        spec = NativeLibrarySpec.from_dict(native_lib_info)
        generated = buck2_generator.generate(spec)

        rules_to_load.extend(generated.rules_to_load)
        deps = deps + generated.extra_deps
        rustc_flags = rustc_flags + generated.extra_rustc_flags

Layout Trait (FUSE Composition)

The composition layer uses a similar pattern for file system layouts.

Layout Trait

#![allow(unused)]
fn main() {
pub trait Layout: Send + Sync {
    /// Get the layout name (e.g., "buck2", "bazel")
    fn name(&self) -> &'static str;

    /// Map a dependency path to its location in the composed view
    fn map_dep(&self, ctx: &LayoutContext, cell: &str, path: &Path) -> Option<PathBuf>;

    /// Generate configuration files for this build system
    fn generate_config(&self, ctx: &LayoutContext) -> Vec<ConfigFile>;

    /// Get the list of cells this layout supports
    fn supported_cells(&self, ctx: &LayoutContext) -> Vec<String>;
}
}

Buck2Layout Implementation

#![allow(unused)]
fn main() {
impl Layout for Buck2Layout {
    fn name(&self) -> &'static str { "buck2" }

    fn generate_config(&self, ctx: &LayoutContext) -> Vec<ConfigFile> {
        vec![
            self.generate_buckconfig(ctx),
            ConfigFile::new(".buckroot", "# Buck2 repository root marker\n"),
        ]
    }

    // ...
}
}

See src/rust/composition/src/layout.rs for the full implementation.

Adding a New Build System

1. Create Native Library Generator

# src/python/buildsystem/newbuild.py
from .native_library import NativeLibrarySpec, GeneratedRules

class NewBuildNativeLibraryGenerator:
    @property
    def name(self) -> str:
        return "newbuild"

    def generate(self, spec: NativeLibrarySpec) -> GeneratedRules:
        # Generate rules for your build system
        lines = [
            f'native_lib(name = "{spec.lib_name}", ...)',
        ]
        return GeneratedRules(
            rules_content="\n".join(lines),
            rules_to_load=["native_lib"],
            extra_deps=[f":{spec.lib_name}"],
            extra_rustc_flags=[f"-Lnative={spec.link_search_path}"],
        )

newbuild_generator = NewBuildNativeLibraryGenerator()

2. Create Layout Implementation (for FUSE)

#![allow(unused)]
fn main() {
// src/rust/composition/src/layouts/newbuild.rs
pub struct NewBuildLayout;

impl Layout for NewBuildLayout {
    fn name(&self) -> &'static str { "newbuild" }

    fn generate_config(&self, ctx: &LayoutContext) -> Vec<ConfigFile> {
        // Generate config files for your build system
        vec![ConfigFile::new("BUILD.newbuild", "# config")]
    }

    // ...
}
}

3. Register the Layout

#![allow(unused)]
fn main() {
// src/rust/composition/src/layout.rs
pub fn layout_by_name(name: &str) -> Option<BoxedLayout> {
    match name {
        "buck2" => Some(Box::new(Buck2Layout::new())),
        "newbuild" => Some(Box::new(NewBuildLayout::new())),
        _ => None,
    }
}
}

Design Principles

1. Specification vs Generation - Keep specifications generic, push build-system details to generators
2. Protocol-based - Use protocols/traits for loose coupling
3. Singleton instances - Generators are stateless, use module-level instances
4. Incremental adoption - New build systems can be added without changing existing code

FUSE Composition Layer

The FUSE composition layer provides a unified filesystem view of repositories and their dependencies. This document covers the architecture for developers extending or maintaining the composition system.

Architecture Overview

┌────────────────────────────────────────────────────────────────┐
│                   CompositionBackend trait                     │
├────────────────────────────────────────────────────────────────┤
│  ┌─────────────────────┐       ┌─────────────────────┐         │
│  │   FUSE Backend      │       │   Symlink Backend   │         │
│  │   (Development)     │       │   (CI / Fallback)   │         │
│  └─────────────────────┘       └─────────────────────┘         │
│              │                           │                     │
│              └───────────┬───────────────┘                     │
│                          ▼                                     │
│              ┌───────────────────────┐                         │
│              │   Composition API     │                         │
│              │   (shared interface)  │                         │
│              └───────────────────────┘                         │
└────────────────────────────────────────────────────────────────┘

Core Components

Backend Trait

The CompositionBackend trait (src/rust/composition/src/backend.rs) defines the interface for all backends:

#![allow(unused)]
fn main() {
pub trait CompositionBackend: Send + Sync {
    fn mount(&mut self) -> Result<()>;
    fn unmount(&mut self) -> Result<()>;
    fn status(&self) -> BackendStatus;
    fn cell_path(&self, cell: &str) -> Option<PathBuf>;
    fn refresh(&mut self) -> Result<()>;
}
}

Backend Selection

The selector (src/rust/composition/src/selector.rs) automatically chooses the appropriate backend:

#![allow(unused)]
fn main() {
pub fn select_backend(requested: BackendType) -> BackendSelection {
    match requested {
        BackendType::Auto => {
            if is_fuse_available() {
                BackendSelection::fuse("Auto-selected FUSE")
            } else {
                BackendSelection::symlink("Auto-selected symlinks (FUSE unavailable)")
            }
        }
        BackendType::Fuse => { /* ... */ }
        BackendType::Symlink => { /* ... */ }
    }
}
}

State Machine

The consistency state machine (src/rust/composition/src/state.rs) manages transitions:

Settled ──manifest change──► Syncing ──nix build──► Building
   ▲                                                    │
   │                                               build done
   │                                                    │
   └───────────────────── Transitioning ◄───────────────┘

Key types:

• ConsistencyStateMachine - Thread-safe state management
• StateObserver - Trait for state change notifications
• CellUpdate - Pending cell updates during transitions

Policy System

The policy system (src/rust/composition/src/policy.rs) controls access during updates. See FUSE Access Policy for details.

Layout System

Layouts (src/rust/composition/src/layout.rs) control how files are presented:

• Layout trait - Core interface for layouts
• LayoutRegistry - Runtime layout registration
• Buck2Layout - Default Buck2 layout
• BazelLayout - Bazel layout

See Custom Layouts for creating new layouts.

Module Structure

src/rust/nix-eval/src/          # Nix client abstraction (replaceable)
├── lib.rs                      # NixClient trait
├── cli.rs                      # CliNixClient (shells out to nix binary)
└── error.rs

src/rust/composition/src/
├── lib.rs                      # Public API exports
├── backend.rs                  # CompositionBackend trait
├── compose_config.rs           # compose.toml parser (single-mount legacy)
├── config.rs                   # CompositionConfig, CellConfig
├── discover.rs                 # Cell discovery via NixClient trait
├── error.rs                    # Error types
├── layout.rs                   # Layout system (Buck2Layout, BazelLayout)
├── policy.rs                   # Access policies
├── recovery.rs                 # Error recovery utilities
├── selector.rs                 # Backend selection logic
├── serve_config.rs             # Service config ([[mounts]] TOML format)
├── service.rs                  # Launchd/systemd service generation
├── state.rs                    # Consistency state machine
├── status.rs                   # BackendStatus enum
├── symlink.rs                  # Symlink backend
├── synthetic.rs                # macOS synthetic firmlink management
├── tracing.rs                  # Logging and debugging
├── watcher.rs                  # File watching (optional)
└── fuse/                       # FUSE backend (feature-gated)
    ├── mod.rs                  # Re-exports FuseBackend (platform-conditional)
    ├── fs_core.rs              # Platform-agnostic filesystem logic
    ├── platform.rs             # Platform detection and FUSE availability
    ├── filesystem.rs           # Linux: fuser Filesystem trait impl
    ├── backend.rs              # Linux: FuseBackend using fuser crate
    ├── edit_overlay.rs         # Copy-on-write editing layer
    ├── patch_generator.rs      # Unified diff generation for edits
    └── fuse_t/                 # macOS: direct libfuse-t backend
        ├── mod.rs
        ├── bindings.rs         # Hand-written FFI bindings to libfuse3
        ├── operations.rs       # FUSE operation callbacks (path-based API)
        └── backend.rs          # FuseTBackend implementing CompositionBackend

nix/home-manager/
└── turnkey-composed.nix        # Home-manager module for service management

Daemon Architecture

The turnkey-composed daemon supports two modes:

• start: Single mount, ad-hoc usage
• serve: Multi-mount service mode, reads config file, watches for changes

In service mode, the daemon:

1. Reads ~/.config/turnkey/composed.toml for mount declarations
2. For each mount: discovers cells via nix-eval crate, builds them, creates the FUSE mount
3. Watches manifest files for dependency changes (triggers cell rebuild)
4. Watches the config file for new/removed mounts (hot-reload)
5. On macOS, manages synthetic firmlinks for mount points under /

Nix Integration

Cell derivations are exposed as flake packages (godeps-cell, rustdeps-cell, etc.) by the flake-parts module. The daemon builds them via the NixClient trait (currently CliNixClient which shells out to nix). This abstraction allows replacing the CLI with a direct Nix daemon client when one becomes available.

FUSE Backend Implementation

The FUSE backend uses a layered architecture with a platform-agnostic core and platform-specific adapters.

FsCore (Platform-Agnostic)

FsCore (fs_core.rs) contains all filesystem logic with zero dependency on the fuser crate:

• Path resolution: resolve_path(path) -> ResolvedPath maps FUSE paths to logical locations (Root, Source, CellPrefix, Cell, VirtualFile, etc.)
• Inode management: Allocation, mapping, and lookup using plain u64 inode numbers
• Virtual file generation: .buckconfig and .buckroot content
• Policy checking: Access control during dependency updates
• Edit overlay: Copy-on-write editing of external dependencies

Both the Linux and macOS backends delegate to FsCore for all filesystem logic, converting between their own FUSE types and FsCore's neutral types.

Linux Backend (fuser crate)

Uses the fuser crate's low-level inode-based API:

• CompositionFs wraps FsCore and implements fuser::Filesystem
• Converts between fuser::INodeNo/FileAttr and FsCore's u64/FsAttr
• Feature flag: fuse (enables dep:fuser)

macOS Backend (FUSE-T FFI)

Uses direct C FFI to FUSE-T's libfuse3, bypassing the fuser crate entirely. This is necessary because fuser reads the FUSE file descriptor directly, which is incompatible with FUSE-T's NFS-based socket protocol.

• bindings.rs: Hand-written FFI bindings to libfuse3 (44-field fuse_operations struct at 352 bytes, fuse_new, fuse_mount, fuse_loop, etc.)
• operations.rs: extern "C" callbacks using the high-level path-based API. Each callback retrieves FsCore via a global AtomicPtr and delegates to resolve_path()
• backend.rs: FuseTBackend spawns a thread calling fuse_new + fuse_mount + fuse_loop
• Feature flag: fuse-t (only dep:libc needed)
• Links against /usr/local/lib/libfuse3.dylib (from FUSE-T)

FUSE-T quirks discovered during implementation:

• fuse_get_context()->private_data does not reliably pass the user_data from fuse_new. A global AtomicPtr<FsCore> is used instead.
• readdir filler must pass null for the stat buffer. FUSE-T's NFS translation rejects certain stat formats with "RPC struct is bad".
• The fuse_operations struct must include the newer statx and syncfs fields even if unused, to match the 352-byte C ABI.

Conditional Compilation

Platform selection happens at compile time:

#![allow(unused)]
fn main() {
// In fuse/mod.rs:
#[cfg(target_os = "linux")]
pub use backend::FuseBackend;           // fuser-based

#[cfg(target_os = "macos")]
pub use fuse_t::backend::FuseTBackend as FuseBackend;  // libfuse-t FFI
}

The selector.rs gates on #[cfg(any(feature = "fuse", feature = "fuse-t"))] so both feature flags enable the FUSE code path.

Platform Detection

Runtime FUSE availability checking in platform.rs:

• Linux: Checks for /dev/fuse
• macOS: Checks for FUSE-T bundle (/Library/Filesystems/fuse-t.fs) or library (/usr/local/lib/libfuse-t.dylib)

Recovery System

The recovery module (src/rust/composition/src/recovery.rs) provides:

Retry Logic

#![allow(unused)]
fn main() {
pub async fn retry_with_backoff<T, F, Fut>(
    config: &RetryConfig,
    operation: F,
) -> Result<T>
where
    F: Fn() -> Fut,
    Fut: Future<Output = Result<T>>,
}

Error Classification

#![allow(unused)]
fn main() {
pub fn is_transient_error(error: &Error) -> bool {
    matches!(error, Error::Timeout(_) | Error::PathUpdating(_) | ...)
}
}

Recovery Actions

#![allow(unused)]
fn main() {
pub enum RecoveryAction {
    Retry { delay: Duration },
    ForceUnmount,
    RestartDaemon,
    ManualIntervention { instructions: String },
}
}

Tracing and Debugging

The tracing module (src/rust/composition/src/tracing.rs) provides:

Configuration

#![allow(unused)]
fn main() {
pub struct TracingConfig {
    pub enable_fuse_ops: bool,
    pub enable_state_transitions: bool,
    pub enable_metrics: bool,
    pub log_level: String,
}
}

State Logger

Implements StateObserver to log state transitions:

#![allow(unused)]
fn main() {
impl StateObserver for StateLogger {
    fn on_state_change(&self, from: SystemState, to: SystemState) {
        info!("State: {:?} -> {:?}", from, to);
    }
}
}

Metrics

Tracks performance metrics:

• Operation counts (lookup, read, readdir, etc.)
• Latency histograms
• Cache hit rates

Debug Information

#![allow(unused)]
fn main() {
pub struct DebugInfo {
    pub backend_type: String,
    pub mount_point: Option<PathBuf>,
    pub cells: Vec<CellDebugInfo>,
    pub state: SystemState,
    pub metrics: Option<Metrics>,
}
}

Testing

Unit Tests

Each module has unit tests:

cargo test -p composition

Integration Tests

Test with actual FUSE mounts (requires FUSE):

# Linux
cargo test -p composition --features fuse -- --ignored

# macOS (FUSE-T)
cargo test -p composition --features fuse-t -- --ignored

Mock Backend

For testing without FUSE:

#![allow(unused)]
fn main() {
use composition::testing::MockBackend;

let backend = MockBackend::new()
    .with_cell("godeps", "/nix/store/xxx-godeps")
    .with_status(BackendStatus::Ready);
}

Feature Flags

The crate uses feature flags:

[features]
default = []
fuse = ["fuser"]       # Enable FUSE backend
watcher = ["notify"]   # Enable file watching

Error Handling

The Error enum in error.rs covers all failure modes:

#![allow(unused)]
fn main() {
pub enum Error {
    AlreadyMounted(PathBuf),
    NotMounted,
    MountPointInaccessible { path, source },
    CellNotFound(String),
    FuseUnavailable(String),
    // ...
}
}

Errors include recovery suggestions:

#![allow(unused)]
fn main() {
impl Error {
    pub fn is_transient(&self) -> bool { /* ... */ }
    pub fn recovery_suggestion(&self) -> Option<String> { /* ... */ }
}
}

Configuration

The CompositionConfig struct holds all settings:

#![allow(unused)]
fn main() {
pub struct CompositionConfig {
    pub mount_point: PathBuf,
    pub cells: HashMap<String, CellConfig>,
    pub consistency_mode: ConsistencyMode,
    pub layout: String,
}

pub struct CellConfig {
    pub source_path: PathBuf,
    pub editable: bool,
}
}

Related Documentation

• FUSE Access Policy - Access control during updates
• Custom Layouts - Creating build system layouts
• Architecture Proposal - Original design document

FUSE Access Policy System

The FUSE composition layer includes a pluggable access policy system that controls how file operations behave during dependency updates. This allows developers to tune the trade-off between consistency and availability based on their workflow.

Overview

When the composition system is updating (rebuilding Nix derivations), file access to dependency cells may need to be controlled. The policy system determines whether to:

• Allow the operation immediately
• Block until the system becomes stable
• Deny with an error (e.g., EAGAIN)
• Allow with stale data and log a warning

┌─────────────────────────────────────────────────────────────┐
│                     FUSE Operation                          │
│   (lookup, getattr, read, readdir, write, create, ...)      │
└─────────────────────────────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────┐
│                   Classify Request                          │
│  path/inode → FileClass                                     │
│  state machine → SystemState                                │
│  operation → OperationType                                  │
└─────────────────────────────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────┐
│                   Policy.check()                            │
│  (FileClass, SystemState, OperationType) → PolicyDecision   │
└─────────────────────────────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────┐
│                   Execute Decision                          │
│  Allow → proceed                                            │
│  Block → wait then retry                                    │
│  Deny → return errno                                        │
│  AllowStale → proceed with warning                          │
└─────────────────────────────────────────────────────────────┘

Core Concepts

File Classes

Files in the composition view are classified by their behavioral characteristics:

Key insight: SourcePassthrough and virtual files are always accessible regardless of system state. Only CellContent access is subject to policy decisions.

System States

The composition system transitions through these states:

Settled ──manifest change──► Syncing ──nix build──► Building
   ▲                                                    │
   │                                               build done
   │                                                    │
   └───────────────────── Transitioning ◄───────────────┘

Operation Types

Built-in Policies

StrictPolicy

Best for: CI pipelines, production builds where correctness is critical

Blocks all cell access during any update phase. Reads will never return stale data, but may block for the duration of the Nix build.

#![allow(unused)]
fn main() {
StrictPolicy::new()                    // Default 5-minute timeout
StrictPolicy::with_timeout(Duration::from_secs(120))  // Custom timeout
}

LenientPolicy

Best for: Interactive development where latency matters

Allows stale reads during syncing and building phases, only blocks during the brief transition phase.

#![allow(unused)]
fn main() {
LenientPolicy::new()
}

CIPolicy

Best for: CI/CD environments where blocking is undesirable

Never blocks - immediately returns EAGAIN if the operation would need to wait. The caller can retry or handle the error.

#![allow(unused)]
fn main() {
CIPolicy::new()
}

DevelopmentPolicy (Default)

Best for: Day-to-day development work

A balanced approach:

• Syncing: Allow stale reads (quick phase)
• Building: Block (wait for fresh data)
• Error: Allow stale (degrade gracefully)

#![allow(unused)]
fn main() {
DevelopmentPolicy::new()
}

Creating Custom Policies

Implement the AccessPolicy trait to create custom behavior:

#![allow(unused)]
fn main() {
use composition::policy::{
    AccessPolicy, FileClass, SystemState, OperationType, PolicyDecision,
};
use std::time::Duration;

pub struct MyPolicy {
    block_timeout: Duration,
}

impl AccessPolicy for MyPolicy {
    fn check(
        &self,
        class: &FileClass,
        state: SystemState,
        op: OperationType,
    ) -> PolicyDecision {
        // Source files always accessible
        if class.is_always_accessible() {
            return PolicyDecision::Allow;
        }

        // Custom logic based on state and operation
        match (state, op) {
            // Allow reads during syncing
            (SystemState::Syncing, OperationType::Read) => {
                PolicyDecision::AllowStale
            }
            // Block lookups during building
            (SystemState::Building, OperationType::Lookup) => {
                PolicyDecision::Block {
                    timeout: self.block_timeout,
                }
            }
            // Fail fast for directory listing during updates
            (_, OperationType::Readdir) if state.is_updating() => {
                PolicyDecision::eagain()
            }
            // Default: allow
            _ => PolicyDecision::Allow,
        }
    }

    fn name(&self) -> &'static str {
        "my-policy"
    }

    fn description(&self) -> &'static str {
        "Custom policy with special handling for readdir"
    }
}
}

Policy Decision Types

Convenience Constructors

#![allow(unused)]
fn main() {
PolicyDecision::block()                    // Block with 5-minute timeout
PolicyDecision::block_with_timeout(dur)    // Block with custom timeout
PolicyDecision::eagain()                   // Deny with EAGAIN (11)
PolicyDecision::ebusy()                    // Deny with EBUSY (16)
}

Configuring the Policy

In Rust Code

When creating a CompositionFs, use the with_policy constructor:

#![allow(unused)]
fn main() {
use composition::{CompositionConfig, CompositionFs};
use composition::policy::{CIPolicy, StrictPolicy};

// With CI policy
let fs = CompositionFs::with_policy(
    config,
    repo_root,
    state_machine,
    Box::new(CIPolicy::new()),
);

// With strict policy and custom timeout
let fs = CompositionFs::with_policy(
    config,
    repo_root,
    state_machine,
    Box::new(StrictPolicy::with_timeout(Duration::from_secs(60))),
);
}

Via Nix Configuration (Future)

turnkey.fuse = {
  enable = true;

  # Policy selection
  accessPolicy = "development";  # "strict" | "lenient" | "ci" | "development"

  # Custom timeout for blocking policies
  blockTimeout = 300;  # seconds
};

Environment Variables (Future)

# Override policy at runtime
TURNKEY_ACCESS_POLICY=ci tk build //...

# Custom timeout
TURNKEY_BLOCK_TIMEOUT=60 tk build //...

Debugging Policies

Policy decisions are logged at debug level. Enable debug logging to see:

DEBUG Policy 'development': blocking for up to 300s until stable
DEBUG Policy 'ci': denying Readdir on CellContent { cell: "godeps" } in state Building
WARN  Policy 'lenient': returning potentially stale data for cell 'godeps' during Building

Guidelines for Choosing a Policy

API Reference

Module: composition::policy

Types:

• FileClass - File classification enum
• SystemState - System state enum
• OperationType - Operation type enum
• PolicyDecision - Decision enum
• AccessPolicy - Policy trait
• BoxedPolicy - Type alias for Box<dyn AccessPolicy>

Built-in Policies:

• StrictPolicy
• LenientPolicy
• CIPolicy
• DevelopmentPolicy

Functions:

• default_policy() - Returns a boxed DevelopmentPolicy

Constants:

• EAGAIN - Resource temporarily unavailable (11)
• EBUSY - Device or resource busy (16)

Flake-Parts Module

Located at nix/flake-parts/turnkey/default.nix.

Purpose

Provides the user-facing API for Turnkey configuration in flakes.

Options Reference

turnkey.toolchains.enable

type = types.bool;
default = true;

Enable/disable Turnkey toolchain management.

turnkey.toolchains.declarationFiles

type = types.attrsOf types.path;
default = {};

Map shell names to toolchain.toml files:

declarationFiles = {
  default = ./toolchain.toml;
  ci = ./toolchain.ci.toml;
};

turnkey.toolchains.registry

type = types.lazyAttrsOf types.package;
default = {};

Custom toolchain registry. If empty, uses default registry.

turnkey.toolchains.wrapNativeTools

type = types.bool;
default = true;

Wrap go, cargo, uv with auto-sync behavior. Every version of the registry's entry is wrapped, so the wrapper runs the tool toolchain.toml resolves.

turnkey.toolchains.buck2

Nested options for Buck2 integration. They are declared once, in nix/buck2/options.nix, which the devenv module uses as well; this module adds only buck2.shells, builds the dependency cells, and hands the whole value to each shell's devenv module. The main ones:

• enable - Enable Buck2 cell generation
• prelude.path - A prelude to use instead of turnkey's (off the supported path; turns test result caching off)
• go.enable, go.depsFile - Go dependency configuration
• rust.enable, rust.depsFile - Rust dependency configuration
• python.enable, python.depsFile - Python dependency configuration

Implementation

The module:

1. Imports default registry
2. Builds tw wrappers for native tools
3. Creates shell configurations for each declaration file
4. Passes configuration to devenv module

Extending

Add new options in the options.perSystem block and implement in config.perSystem.

Devenv Module

Located at nix/devenv/turnkey/default.nix.

Purpose

Configures individual devenv shells with toolchains and Buck2 integration.

Options

turnkey.enable

Enable Turnkey for this shell.

turnkey.declarationFile

Path to toolchain.toml file.

turnkey.registry

Package registry (usually inherited from flake-parts).

How It Works

1. Parse TOML: Reads toolchain.toml

toolchainDeclaration = builtins.fromTOML (builtins.readFile cfg.declarationFile);
toolchainNames = builtins.attrNames toolchainDeclaration.toolchains;

2. Resolve packages: Maps names to packages

resolvedPackages = map (name: cfg.registry.${name}) toolchainNames;

3. Add to shell: Packages added to devenv

config.packages = resolvedPackages;

Sub-Module: buck2.nix

The buck2.nix sub-module (nix/devenv/turnkey/buck2.nix) handles:

• Toolchains cell symlink (the flake-parts module builds the cell, see Buck2 Cells)
• Prelude cell symlink
• Dependency cell symlinks
• Shell entry hooks

Shell Entry Hooks

Devenv's enterShell hook:

1. Symlinks .turnkey/prelude → Nix store
2. Symlinks .turnkey/toolchains → Nix store
3. Symlinks dependency cells if configured
4. Displays welcome message

Debugging

Enable verbose output:

TURNKEY_VERBOSE=1 nix develop

Buck2 Cell Generation

This document describes how Turnkey generates Buck2 cells from Nix derivations.

Overview

Turnkey generates several types of Buck2 cells:

• Toolchains cell - Language toolchains (Go, Rust, Python, etc.)
• Dependency cells - Third-party packages (godeps, rustdeps, pydeps)
• Prelude cell - Buck2 prelude with extensions

All cells are built as Nix derivations and symlinked into .turnkey/.

Toolchains Cell

Located at nix/buck2/toolchains-cell.nix. Generated from nix/buck2/mappings.nix.

nix/buck2/toolchains-cell-package.nix builds it for a shell's toolchain.toml. The flake-parts module builds one per shell, hands it to the shell (turnkey.buck2.toolchainsCell), which symlinks it at .turnkey/toolchains, and exposes the default shell's as the toolchains-cell package. The composition daemon builds that package like every other *-cell.

Mapping Structure

{
  go = {
    skip = false;
    targets = [{
      name = "go";
      rule = "system_go_toolchain";
      load = "@prelude//toolchains/go:system_go_toolchain.bzl";
      visibility = [ "PUBLIC" ];
      dynamicAttrs = registry: {
        go_binary = "${registry.go}/bin/go";
      };
    }];
    implicitDependencies = [ "python" "cxx" ];
    runtimeDependencies = [ ];
  };
}

Generated Output

rules.star is generated with:

1. Load statements for each rule
2. Rule instantiations with configured attributes
3. Visibility set to PUBLIC

Adding Toolchain Mappings

Edit nix/buck2/mappings.nix:

mylang = {
  skip = false;
  targets = [{
    name = "mylang";
    rule = "system_mylang_toolchain";
    load = "@prelude//mylang:toolchain.bzl";
    visibility = [ "PUBLIC" ];
    dynamicAttrs = registry: {
      compiler_path = "${registry.mylang}/bin/mylang";
    };
  }];
  implicitDependencies = [ ];
  runtimeDependencies = [ ];
};

Go Dependency Cell

Built by the Go adapter, nix/lib/deps-cell/adapters/go.nix. Like the Rust cell, it is a write-once cell (ADR 0004, ADR 0008): a real directory in the project, with one store link per Go module and one alias package per Go package.

Cell Structure

.turnkey/godeps/
├── .buckconfig                       # Cell identity
├── .deps-file-sha256                 # the go-deps.toml it was built from
├── _store/
│   └── <hash>-dep-go-golang-org-x-sys -> /nix/store/<hash>-dep-go-golang-org-x-sys
│       ├── cpu/rules.star            # go_library, generated in the module's derivation
│       └── unix/rules.star
└── vendor/
    └── golang.org/x/sys/
        ├── cpu/rules.star            # alias -> //_store/<hash>-dep-go-golang-org-x-sys/cpu:cpu
        └── unix/rules.star           # alias -> //_store/<hash>-dep-go-golang-org-x-sys/unix:unix

The vendor/ tree mirrors Go import paths, so labels name the import path and never a version. Alias packages nest where modules do (vendor/cloud.google.com/go and vendor/cloud.google.com/go/storage).

Process

1. tk sync runs godeps-gen, which records each module's version and hash in go-deps.toml, and the go.work members under [members].
2. Nix builds one package per module (mkGoDepPackage): the module's source from the Go proxy, its fixup, its user patches (.turnkey/patches/godeps/vendor/<module path>/), the assembly headers its .s files include, and its rules.star files, generated by buckgen from the module's own source and the cell-wide settings (cell name, platforms, allowed build tags, Go minor version). Its targets output lists the Go packages rendered (<subdir> <target>), its imports output the import paths they reference.
3. mkCellIndex builds the cell index: one package per Go package, at vendor/<import path>, with its module's store path and its subdir there. An import path two modules offer goes to the longer module path. A referenced import path a go.work member owns becomes a forwarding alias package, to the member's target in the root cell (root//<member dir>/<rest>:<last component>).
4. The shell runs tk materialize with the index, as for the Rust cell.

A module bump rebuilds that module's derivation alone, and buck2 recompiles its reverse dependencies alone. Adding or removing a go.work member rebuilds no module: only the index and its forwarding aliases change.

Cell Configuration

# .buckconfig
[cells]
    godeps = .
    prelude = bundled://

[buildfile]
    name = rules.star

Generated rules.star Files

Each Go package a configured platform builds gets a go_library target, in its module's store path:

# _store/<hash>-dep-go-github-com-spf13-cobra/rules.star
go_library(
    name = "cobra",
    package_name = "github.com/spf13/cobra",
    srcs = native.glob(["*.go", "*.s", "*.h", "*.c", "*.cc", "*.cpp", "*.S"]),
    header_namespace = "",
    visibility = ["PUBLIC"],
    deps = [
        "godeps//vendor/github.com/inconshreveable/mousetrap:mousetrap",
        "godeps//vendor/github.com/spf13/pflag:pflag",
    ],
)

Deps that only some platforms or build tags need are a select().

Target Path Format

When writing rules.star files that depend on packages from the godeps cell:

godeps//vendor/<import-path>:<target-name>

Where:

• godeps// - the cell alias (configured in .buckconfig)
• vendor/ - required prefix - all packages live under vendor/
• <import-path> - the full Go import path
• <target-name> - the directory name (last path component), NOT the package name

Examples:

Common Mistakes:

# WRONG - missing vendor/ prefix
deps = ["godeps//github.com/spf13/cobra:cobra"]

# WRONG - using package name instead of directory name for versioned imports
deps = ["godeps//vendor/github.com/pelletier/go-toml/v2:go-toml"]

# CORRECT
deps = ["godeps//vendor/github.com/pelletier/go-toml/v2:v2"]

Import Path Resolution

The go_library rule's package_name makes the Go compiler see the import path, whatever the target's label:

• Buck2 target: godeps//vendor/github.com/spf13/cobra:cobra
• Go import: import "github.com/spf13/cobra"

Rust Dependency Cell

Built by the Rust adapter, nix/lib/deps-cell/adapters/rust.nix. Like the Go cell, it is a write-once cell (ADR 0004): a real directory in the project, not a symlink to one store path.

Process

1. tk sync runs rustdeps-gen, which records each crate's hash and its package slice in rust-deps.toml: features and resolved dependencies as cargo's feature resolver gives them, per platform (ADR 0006).
2. Nix builds one package per crate (mkRustCrates): the crate's source, its fixup, its user patches, and its rules.star, generated by rust-rules-gen from the crate's own slice and fixup alone. A second output lists the crate's target names.
3. mkCellIndex builds the cell index: each package's path, store path and targets, the unversioned aliases (highest version), the cell's .buckconfig, and the SHA-256 of rust-deps.toml. The flake exports it as rustdeps-index.
4. The shell runs tk materialize with the index: it keeps .turnkey/rustdeps in line, with a store link per package under _store/, never retargeted, and an alias package per versioned and unversioned name under vendor/, rewritten only when it changes.

Special Handling

Rust crates may require:

• rustc flags - Build scripts that emit cargo:rustc-cfg
• Generated files - Build scripts that generate .rs files
• Native code - Build scripts that compile C/assembly

See Dependency Generators for handling these cases.

Python Dependency Cell

Built by the Python adapter, nix/lib/deps-cell/adapters/python.nix. Like the Rust and Go cells, it is a write-once cell (ADR 0004, ADR 0010): one store link per distribution, and one alias package per distribution.

Cell Structure

.turnkey/pydeps/
├── .buckconfig                       # Cell identity
├── .deps-file-sha256                 # the python-deps.toml it was built from
├── _store/
│   └── <hash>-dep-python-six-1.17.0 -> /nix/store/<hash>-dep-python-six-1.17.0
│       ├── six.py                    # the unpacked wheel: .py files are srcs,
│       ├── six-1.17.0.dist-info/     # every other file a resource
│       └── rules.star                # python_library, generated in the distribution's derivation
└── vendor/
    └── six/rules.star                # alias -> //_store/<hash>-dep-python-six-1.17.0:six

Labels are pydeps//vendor/<name>:<name>, with <name> the distribution's key in python-deps.toml. python-deps.toml holds one version per name (pydeps-gen fails on a forked lock), so there are no version alias packages.

Process

1. tk sync runs pydeps-gen, which records each distribution's version, and its pure (py3-none-any) wheel's URL and unpacked hash, from pylock.toml (python-deps.toml schema 3), and its dependencies, markers and extras from uv.lock. A distribution with no pure wheel fails it (ADR 0013).
2. Nix builds one package per distribution (mkPythonDepPackage): its wheel from PyPI, unpacked and installed (installWheel: <name>.data/'s purelib and platlib merged into the root, the rest dropped), its fixup, its user patches (.turnkey/patches/pydeps/vendor/<name>/), and its rules.star, written by pydeps-cell from the distribution's package slice (its dependencies and its requested extras', narrowed by sliceOf to the distributions the cell holds), the platforms' conditions and the Python toolchain's version. Its targets output holds its one target, <name>.
3. mkCellIndex builds the cell index: one package per distribution, at vendor/<name>, with its store path.
4. The shell runs tk materialize with the index, as for the Rust cell.

A version bump rebuilds that distribution's derivation, and those whose slice names it only if their slice changed; buck2 re-runs its reverse dependencies alone.

Solidity Dependency Cell

Built by the Solidity adapter, nix/lib/deps-cell/adapters/solidity.nix. Like the Go and Rust cells, it is a write-once cell (ADR 0004, ADR 0011), and the first whose users address its root package.

Cell Structure

.turnkey/soldeps/
├── .buckconfig
├── .deps-file-sha256                 # the solidity-deps.toml it was built from
├── rules.star                        # bundle and one alias per package, from the index's root
├── _store/
│   └── <hash>-dep-sol-forge_std-1.8.0 -> /nix/store/<hash>-dep-sol-forge_std-1.8.0
│       └── rules.star                # forge_std (.sol files) and forge_std_all filegroups
└── vendor/
    └── forge-std/
        ├── rules.star                # alias -> //_store/<hash>-dep-sol-forge_std-1.8.0:forge_std(_all)
        └── src -> /nix/store/<hash>-dep-sol-forge_std-1.8.0/src   # for native forge

Process

1. tk sync runs soldeps-gen, which writes one [[package]] per name to solidity-deps.toml (a name declared twice resolves to one package, or fails), and the root remappings.txt.
2. Nix builds one package per Solidity package (mkSolDepPackage): the npm tarball or git archive, its fixup, its user patches (.turnkey/patches/soldeps/vendor/<name>/), and its rules.star. Its targets output lists <target> and <target>_all.
3. mkCellIndex builds the cell index: one package per Solidity package at vendor/<name>, with no version aliases, and root, the root package's rules.star: bundle, which maps each vendor/<name> to //vendor/<name>:<target>_all, and an alias per package. Packages are exposed.
4. The shell runs tk materialize with the index. It writes root as it is, as it writes .buckconfig, and links each exposed package's store entries beside its alias package's rules.star. Native forge reads the cell in place through the root remappings.txt, so it needs the files at vendor/<name>/. Those links follow a bump to the new store path; buck2 never reads through them, since the alias package globs nothing.

A bump rebuilds the bumped package's derivation and the index. Every Solidity action stages the whole bundle, so they all re-run, and no action of another language does.

JavaScript Dependency Cell

Built by the JavaScript adapter, nix/lib/deps-cell/adapters/javascript.nix, as a write-once cell that separates a package's contents from where it sits in the graph (ADR 0012).

Cell Structure

.turnkey/jsdeps/
├── .buckconfig
├── .deps-file-sha256                 # the js-deps.toml it was built from
├── rules.star                        # the instance graph, from the index's root
├── _store/
│   └── <hash>-dep-js-micromatch-4.0.8 -> /nix/store/<hash>-dep-js-micromatch-4.0.8
│       └── rules.star                # files: a filegroup of the package
└── vendor/
    └── micromatch@4.0.8/
        └── rules.star                # alias files -> //_store/<hash>-dep-js-micromatch-4.0.8:files

The root rules.star loads @prelude//typescript:npm.bzl (nix/buck2/prelude-extensions/typescript/npm.bzl). The rules live in the prelude, not in the cell: a transitive set's type must be defined once.

• npm_instance, one per [[instance]], named after pnpm's node_modules/.pnpm directory for its key (micromatch@4.0.8, react-dom@18.2.0_react@18.2.0), cut and hashed past 240 bytes. Its deps are by import name, and its optional deps that install on some platforms only are a select().
• npm_component, one per dependency cycle (a strongly connected component of instances, computed at evaluation), and an npm_member forwarding to it per instance in the cycle.
• alias, one per [direct] entry, named by the npm name (@types/micromatch). Instances are private to the cell's root package.

An instance's action copies its files with every link dereferenced into node_modules/<name>/, and links each dependency beside it, relative to its own output, into the dependency's. The output has no content-based path, and the links are passed with ignore_artifacts, so a link is keyed by its path: a dependency's content change re-runs only what copies it. NpmPackageInfo carries the package directory and the transitive set of instance outputs. typescript_library and typescript_binary link their npm_deps into a symlinked_dir node_modules and carry that set as hidden inputs (compile.bzl).

Process

1. tk sync runs jsdeps-gen, which writes [[package]] (one per name@version), [[instance]] (one per pnpm snapshot, with its dependencies resolved to instance keys) and [direct].
2. Nix builds one package per [[package]] (mkJsDepPackage): the npm tarball (or the project's own copy, buck2.javascript.tarballs), its fixup, its user patches (.turnkey/patches/jsdeps/vendor/<name>@<version>/), and its rules.star. Its targets output lists files.
3. mkCellIndex builds the cell index: one package per [[package]] at vendor/<name>@<version>, and root, the instance graph.
4. The shell runs tk materialize with the index.

A bump rebuilds the bumped package's derivation and the index, and re-runs the instances that link it, transitively, and their consumers.

Cell Configuration

Each cell gets a .buckconfig:

[cells]
    cellname = .
    prelude = path/to/prelude

[buildfile]
    name = rules.star

Dual-Build Compatibility

A key design goal is that code builds with both native tools and Buck2:

# Native build (uses go.mod directly)
go build ./...

# Buck2 build (uses generated cell)
buck2 build //...

This is achieved by:

1. No import rewriting - Go code uses standard import paths (github.com/foo/bar)
2. importpath attribute - Buck2's go_library rule's importpath tells the compiler the correct path
3. Nix-managed deps - Dependencies fetched by Nix, not vendored in repo

Adding New Dependency Cell Types

To add support for a new language:

1. Create deps generator (e.g., newlang-deps-gen)
2. Create cell builder (an adapter in nix/lib/deps-cell/adapters/)
3. Add the language's record to nix/buck2/languages.nix
4. Add configuration options to nix/buck2/options.nix

Debugging

Inspect Generated rules.star Files

cat .turnkey/godeps/vendor/github.com/spf13/cobra/rules.star          # the alias
cat .turnkey/godeps/_store/*-dep-go-github-com-spf13-cobra/rules.star  # the go_library

Check Cell Contents

ls -la .turnkey/godeps/vendor/

Verify Cell Configuration

cat .turnkey/godeps/.buckconfig

List Available Targets

buck2 targets godeps//...

Adding Toolchains

This guide covers adding new toolchains to Turnkey.

Steps

1. Add package to registry (versioned format)
2. Add mapping to mappings.nix
3. (Optional) Create prelude extension

1. Add to Registry

For standard toolchains (available in nixpkgs), contribute to teller's registry/default.nix.

For project-specific tools, use registryExtensions in your flake.nix:

turnkey.toolchains = {
  registryExtensions = let
    single = pkg: { versions = { "default" = pkg; }; default = "default"; };
  in {
    # Single version (most common)
    zig = single pkgs.zig;

    # Multiple versions
    nodejs = {
      versions = {
        "18" = pkgs.nodejs_18;
        "20" = pkgs.nodejs_20;
        "22" = pkgs.nodejs_22;
      };
      default = "20";
    };
  };
};

Versioned Format

Each registry entry must have:

<name> = {
  versions = { "<version>" = <derivation>; ... };
  default = "<version>";  # Must match a key in versions
};

The single helper is convenient for tools with only one version.

2. Add Toolchain Mapping

Edit nix/buck2/mappings.nix:

For Standard Toolchains

zig = {
  skip = false;
  targets = [{
    name = "zig";
    rule = "system_zig_toolchain";
    load = "@prelude//zig:toolchain.bzl";
    visibility = [ "PUBLIC" ];
  }];
  implicitDependencies = [ ];
};

For Non-Toolchain Tools

Some tools don't need Buck2 rules:

mydevtool = {
  skip = true;
  reason = "Development utility, not a Buck2 toolchain";
};

3. Dynamic Attributes

For toolchains needing Nix store paths, use dynamicAttrs. The function receives a resolved registry where entries are already derivations:

mylang = {
  targets = [{
    name = "mylang";
    rule = "system_mylang_toolchain";
    load = "@prelude//mylang:toolchain.bzl";
    # registry entries are already resolved to derivations
    dynamicAttrs = registry: {
      compiler = "${registry.mylang}/bin/mycompiler";
    };
  }];
};

4. Prelude Extension (if needed)

If the upstream prelude doesn't have rules for your language, create a prelude extension. See Prelude Extensions.

Testing

1. Add toolchain to toolchain.toml
2. Stage files: git add nix/
3. Enter shell: nix develop
4. Verify: tk targets toolchains//...

Creating Custom Registries

For reusable toolchain collections, create a registry flake:

# my-registry/flake.nix
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    turnkey.url = "github:firefly-engineering/turnkey";
  };

  outputs = { nixpkgs, turnkey, ... }:
    let
      forAllSystems = f: nixpkgs.lib.genAttrs
        [ "x86_64-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin" ]
        (system: f system);
    in {
      overlays.default = forAllSystems (system:
        turnkey.lib.${system}.mkRegistryOverlay (final: prev: {
          # Add toolchains
          zig = {
            versions = {
              "0.11" = final.zig_0_11;
              "0.12" = final.zig_0_12;
            };
            default = "0.12";
          };

          # Extend existing toolchain with more versions
          go = {
            versions = { "1.24" = final.go_1_24; };
            default = "1.24";
          };
        })
      );
    };
}

Consumers compose the overlay:

pkgs = import nixpkgs {
  overlays = [
    my-registry.overlays.default.${system}
  ];
};

When multiple registries are composed:

• Versions merge additively - all versions from all registries are available
• Default is overridden - later overlays set the default version

Prelude Extensions

This document covers how to add custom Buck2 rules to the prelude and the various customization approaches available.

Overview

The Buck2 prelude is a collection of Starlark rules that provide build functionality for various languages (Go, Rust, Python, C++, etc.). It's the "standard library" of build rules that ships with Buck2.

Prelude extensions live in nix/buck2/prelude-extensions/. They're copied into the prelude during the Nix build.

Directory Structure

nix/buck2/prelude-extensions/
└── mylang/
    ├── providers.bzl     # Provider definitions
    ├── toolchain.bzl     # Toolchain rule
    ├── mylang_library.bzl
    ├── mylang_binary.bzl
    └── mylang.bzl        # Convenience exports

Creating an Extension

1. Create Provider

providers.bzl:

MylangToolchainInfo = provider(
    doc = "Mylang toolchain information.",
    fields = {
        "compiler": provider_field(typing.Any, default = None),
    },
)

MylangLibraryInfo = provider(
    doc = "Information about a mylang library.",
    fields = {
        "output": provider_field(typing.Any, default = None),
    },
)

2. Create Toolchain Rule

toolchain.bzl:

load(":providers.bzl", "MylangToolchainInfo")

def _system_mylang_toolchain_impl(ctx):
    compiler_path = ctx.attrs.compiler_path
    return [
        DefaultInfo(),
        MylangToolchainInfo(
            compiler = RunInfo(args = cmd_args(compiler_path)),
        ),
    ]

system_mylang_toolchain = rule(
    impl = _system_mylang_toolchain_impl,
    attrs = {
        "compiler_path": attrs.string(
            doc = "Path to the mylang compiler binary",
        ),
    },
    is_toolchain_rule = True,
    doc = "System-provided mylang toolchain.",
)

3. Create Build Rules

mylang_binary.bzl:

load(":providers.bzl", "MylangToolchainInfo")

def _mylang_binary_impl(ctx):
    toolchain = ctx.attrs._toolchain[MylangToolchainInfo]
    out = ctx.actions.declare_output(ctx.label.name)

    ctx.actions.run(
        cmd_args(
            toolchain.compiler.args,
            ctx.attrs.srcs,
            "-o",
            out.as_output(),
        ),
        category = "mylang_compile",
        identifier = ctx.label.name,
    )

    return [
        DefaultInfo(default_output = out),
        RunInfo(args = cmd_args(out)),
    ]

mylang_binary = rule(
    impl = _mylang_binary_impl,
    attrs = {
        "srcs": attrs.list(
            attrs.source(),
            doc = "Source files to compile",
        ),
        "_toolchain": attrs.toolchain_dep(
            default = "toolchains//:mylang",
            providers = [MylangToolchainInfo],
        ),
    },
    doc = "Build a mylang executable.",
)

4. Export Rules

mylang.bzl:

load(":mylang_binary.bzl", _mylang_binary = "mylang_binary")
load(":mylang_library.bzl", _mylang_library = "mylang_library")
load(":toolchain.bzl", _system_mylang_toolchain = "system_mylang_toolchain")
load(":providers.bzl", _MylangToolchainInfo = "MylangToolchainInfo")

mylang_binary = _mylang_binary
mylang_library = _mylang_library
system_mylang_toolchain = _system_mylang_toolchain
MylangToolchainInfo = _MylangToolchainInfo

5. Add Toolchain Mapping

Edit nix/buck2/mappings.nix:

mylang = {
  skip = false;
  targets = [{
    name = "mylang";
    rule = "system_mylang_toolchain";
    load = "@prelude//mylang:toolchain.bzl";
    visibility = [ "PUBLIC" ];
    dynamicAttrs = registry: {
      compiler_path = "${registry.mylang}/bin/mylang";
    };
  }];
  implicitDependencies = [ ];
  runtimeDependencies = [ ];
};

Building

Extensions are included when you rebuild the prelude:

git add nix/buck2/prelude-extensions/mylang/
nix build .#turnkey-prelude

Customization Approaches

Approach 1: Extension Cell Pattern

Create a separate cell for custom rules alongside the standard prelude:

project/
├── prelude/           # Standard prelude (submodule or external)
├── prelude-custom/    # Custom extensions
│   ├── BUCK
│   ├── platforms/
│   ├── toolchains/
│   └── rules/
└── .buckconfig

[cells]
prelude = prelude
prelude-custom = prelude-custom

[external_cells]
prelude = bundled

[build]
execution_platforms = prelude-custom//platforms:default

Pros:

• Clean separation of concerns
• Can still use bundled prelude for core rules
• Easy to track what's custom vs standard
• No fork maintenance burden

Cons:

• Two cells to manage
• Must understand which rules come from where

Approach 2: Custom Rules Outside Prelude

Define rules anywhere in your project - they don't need to be in the prelude:

# rules/my_rules.bzl
def my_custom_rule_impl(ctx):
    # Implementation
    pass

my_custom_rule = rule(
    impl = my_custom_rule_impl,
    attrs = {
        "src": attrs.source(),
        "deps": attrs.list(attrs.dep()),
    },
)

# BUCK
load("//rules:my_rules.bzl", "my_custom_rule")

my_custom_rule(
    name = "my_target",
    src = "input.txt",
)

Pros:

• No prelude modification needed
• Explicit load() makes dependencies clear
• Rules live with the project

Cons:

• Must use explicit load() statements
• Not globally available like prelude rules

Approach 3: Nix-Backed Prelude Cell (Recommended)

This is Turnkey's recommended approach. The prelude Nix derivation:

1. Fetches upstream prelude from buck2-prelude repository
2. Applies turnkey patches for customizations
3. Adds custom rules from nix/buck2/prelude-extensions/

# nix/buck2/prelude.nix
{ pkgs, lib }:

let
  upstreamPrelude = pkgs.fetchFromGitHub {
    owner = "facebook";
    repo = "buck2-prelude";
    rev = "...";  # Pinned commit
    hash = "sha256-...";
  };
in
pkgs.runCommand "turnkey-prelude" {} ''
  cp -r ${upstreamPrelude} $out
  chmod -R u+w $out

  # Apply turnkey patches
  patch -d $out -p1 < ${../patches/prelude/nix-integration.patch}

  # Add custom rules
  cp -r ${./prelude-extensions}/* $out/
''

Advantages:

Approach 4: Forked Prelude

Maintain a fork of the Buck2 prelude with your modifications.

[external_cells]
prelude = git

[external_cell_prelude]
git_origin = https://github.com/your-org/buck2-prelude-fork.git
commit_hash = your-fork-commit-hash

Pros:

• Complete control over all rules
• Can modify any prelude behavior

Cons:

• Significant maintenance burden
• Must track upstream changes
• Risk of divergence from upstream

Prelude Version Compatibility

The Buck2 binary and its prelude must be version-matched. Using a mismatched prelude can cause cryptic Starlark errors.

Symptoms

How turnkey keeps them matched

Turnkey ships one pinned buck2 release (ADR 0002). Its record in nix/buck2/buck2-source.nix names the release date, which is the version key of both toolbox's buck2 and buck2-prelude. It also holds the prelude commit the release was built with, taken from the release's prelude_hash asset. Evaluation fails if toolbox's prelude for that date is a different commit, so a mismatched pair can't be built. Fix a wrong entry in toolbox, not in turnkey.

The binary, the prelude and turnkey's patches for it (nix/patches/prelude/*.patch) move together, in the change that bumps the pin.

Existing Extensions

Turnkey includes these prelude extensions:

• typescript/ - TypeScript compiler integration
• mdbook/ - Documentation builder

When to Customize

Consider prelude customization when:

1. Built-in rules don't support your workflow - e.g., Nix-specific build patterns
2. You need enhanced toolchain control - beyond what system toolchains provide
3. Platform definitions need modification - custom constraint values
4. You're integrating with external systems - CI/CD, remote execution

References

• Buck2 External Cells Documentation
• Buck2 Writing Rules
• Buck2 Writing Toolchains
• Buck2 Prelude Repository

Custom Rules

Create custom Buck2 rules for your project.

Project-Local Rules

For rules specific to your project, create a rules/ directory:

my-project/
├── rules/
│   ├── myrule.bzl
│   └── rules.star
└── rules.star

In rules/rules.star:

# Export rules from this cell
load(":myrule.bzl", "my_rule")

Reference from your project:

load("//rules:myrule.bzl", "my_rule")

my_rule(
    name = "example",
    # ...
)

Prelude-Level Rules

For rules you want available across multiple projects, add them as prelude extensions. See Prelude Extensions.

Rule Structure

Basic Rule

def _my_rule_impl(ctx: AnalysisContext) -> list[Provider]:
    out = ctx.actions.declare_output("output.txt")

    ctx.actions.run(
        cmd_args("echo", "hello", ">", out.as_output()),
        category = "my_rule",
    )

    return [DefaultInfo(default_output = out)]

my_rule = rule(
    impl = _my_rule_impl,
    attrs = {
        "srcs": attrs.list(attrs.source()),
    },
)

With Toolchain

def _my_rule_impl(ctx):
    toolchain = ctx.attrs._toolchain[MyToolchainInfo]
    # Use toolchain...

my_rule = rule(
    impl = _my_rule_impl,
    attrs = {
        "_toolchain": attrs.toolchain_dep(
            default = "toolchains//:mytool",
            providers = [MyToolchainInfo],
        ),
    },
)

With RunInfo

def _my_binary_impl(ctx):
    out = ctx.actions.declare_output(ctx.label.name)
    # Build the binary...

    run_info = RunInfo(args = cmd_args(out))

    return [
        DefaultInfo(default_output = out),
        run_info,  # Makes it runnable with `buck2 run`
    ]

Best Practices

1. Use categories in ctx.actions.run() for build output
2. Declare all outputs explicitly
3. Use hidden deps for non-output dependencies
4. Provide sensible defaults for optional attrs

Custom Layouts

The composition layer uses a pluggable layout system to support different build systems. While Turnkey ships with Buck2 and Bazel layouts, you can create custom layouts for other build systems or specialized requirements.

Layout Architecture

Layouts control how the composed filesystem presents dependencies:

┌─────────────────────────────────────────────────────────────────┐
│                   Layout System                                 │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  LayoutContext ──────────────► Layout.map_dep()                 │
│  (mount point,                   │                              │
│   repo root,                     ▼                              │
│   cells)            /firefly/project/external/godeps/vendor/... │
│                                                                 │
│  LayoutContext ──────────────► Layout.generate_config()         │
│                                  │                              │
│                                  ▼                              │
│                           .buckconfig, .buckroot, etc.          │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

The Layout Trait

All layouts implement the Layout trait:

#![allow(unused)]
fn main() {
use composition::layout::{Layout, LayoutContext, ConfigFile, CellInfo};
use std::path::{Path, PathBuf};

pub trait Layout: Send + Sync {
    /// Layout name (e.g., "buck2", "bazel", "custom")
    fn name(&self) -> &'static str;

    /// Map a dependency path to its composed location
    fn map_dep(&self, ctx: &LayoutContext, cell: &str, path: &Path) -> Option<PathBuf>;

    /// Generate configuration files for this build system
    fn generate_config(&self, ctx: &LayoutContext) -> Vec<ConfigFile>;

    /// List of cells this layout supports
    fn supported_cells(&self, ctx: &LayoutContext) -> Vec<String>;
}
}

LayoutContext

The LayoutContext provides all information needed for layout operations:

#![allow(unused)]
fn main() {
pub struct LayoutContext {
    /// Mount point (e.g., "/firefly/turnkey")
    pub mount_point: PathBuf,

    /// Repository root (actual filesystem path)
    pub repo_root: PathBuf,

    /// Name of the source overlay directory (default: "root")
    pub source_dir_name: String,

    /// Prefix for cell directories (default: "external")
    pub cell_prefix: String,

    /// Available cells
    pub cells: Vec<CellInfo>,
}

pub struct CellInfo {
    /// Cell name (e.g., "godeps")
    pub name: String,

    /// Source path (Nix store path)
    pub source_path: PathBuf,

    /// Whether editing is enabled
    pub editable: bool,
}
}

Helper Methods

#![allow(unused)]
fn main() {
impl LayoutContext {
    /// Get the path to a cell's directory
    /// e.g., "/firefly/turnkey/external/godeps"
    pub fn cell_path(&self, cell: &str) -> PathBuf;

    /// Get the root source directory path
    /// e.g., "/firefly/turnkey/root"
    pub fn source_path(&self) -> PathBuf;

    /// Check if a cell exists
    pub fn has_cell(&self, name: &str) -> bool;
}
}

Creating a Custom Layout

Basic Example

#![allow(unused)]
fn main() {
use composition::layout::{Layout, LayoutContext, ConfigFile};
use std::path::{Path, PathBuf};

pub struct PleaseLayout;

impl Layout for PleaseLayout {
    fn name(&self) -> &'static str {
        "please"
    }

    fn map_dep(&self, ctx: &LayoutContext, cell: &str, path: &Path) -> Option<PathBuf> {
        // Map cell paths to Please's third_party structure
        // e.g., godeps -> third_party/go
        let target_dir = match cell {
            "godeps" => "third_party/go",
            "rustdeps" => "third_party/rust",
            "pydeps" => "third_party/python",
            _ => return None,
        };
        Some(ctx.mount_point.join(target_dir).join(path))
    }

    fn generate_config(&self, ctx: &LayoutContext) -> Vec<ConfigFile> {
        // Generate .plzconfig at the root
        let config = format!(
            r#"[please]
version = >=17.0.0

[build]
path = {}

[go]
importpath = github.com/example/project
"#,
            ctx.source_path().display()
        );

        vec![ConfigFile::new(".plzconfig", config)]
    }

    fn supported_cells(&self, ctx: &LayoutContext) -> Vec<String> {
        ctx.cells
            .iter()
            .filter(|c| matches!(c.name.as_str(), "godeps" | "rustdeps" | "pydeps"))
            .map(|c| c.name.clone())
            .collect()
    }
}
}

Using SimpleLayout

For quick prototyping, use SimpleLayout without implementing the full trait:

#![allow(unused)]
fn main() {
use composition::layout::{SimpleLayout, LayoutContext, ConfigFile};

let layout = SimpleLayout::new(
    "pants",
    |ctx, cell, path| {
        // Custom path mapping
        Some(ctx.mount_point.join("3rdparty").join(cell).join(path))
    },
    |ctx| {
        // Generate config files
        vec![
            ConfigFile::new("pants.toml", "[GLOBAL]\npants_version = \"2.18.0\""),
            ConfigFile::new("BUILD", "# Root BUILD file"),
        ]
    },
);
}

Registering Custom Layouts

Using the Global Registry

Register your layout at application startup:

#![allow(unused)]
fn main() {
use composition::layout::{global_registry, BoxedLayout};

fn register_layouts() {
    let registry = global_registry();
    registry.register(Box::new(PleaseLayout));
    registry.register(Box::new(PantsLayout::new()));
}
}

Using a Custom Registry

For more control, create your own registry:

#![allow(unused)]
fn main() {
use composition::layout::{LayoutRegistry, BoxedLayout};

let mut registry = LayoutRegistry::new();
registry.register(Box::new(PleaseLayout));
registry.register(Box::new(CustomLayout::with_options(opts)));

// Look up by name
let layout = registry.get("please").expect("layout not found");

// List available layouts
for name in registry.available() {
    println!("Layout: {}", name);
}
}

ConfigFile

Generated configuration files use the ConfigFile struct:

#![allow(unused)]
fn main() {
pub struct ConfigFile {
    /// Relative path within the composed view
    pub path: PathBuf,

    /// File content
    pub content: String,
}

impl ConfigFile {
    pub fn new(path: impl Into<PathBuf>, content: impl Into<String>) -> Self;
}
}

Common patterns:

#![allow(unused)]
fn main() {
// Root config file
ConfigFile::new(".buckconfig", "...")

// Nested path
ConfigFile::new("build/config.bzl", "...")

// Per-cell config
ConfigFile::new(format!("{}/{}/BUILD", ctx.cell_prefix, cell), "...")
}

Layout Selection

Layouts are selected via configuration:

Nix Configuration

turnkey.fuse = {
  enable = true;
  layout = "please";  # Use custom layout
};

Runtime Selection

#![allow(unused)]
fn main() {
use composition::layout::{layout_by_name, default_layout};

// Get specific layout
let layout = layout_by_name("please")?;

// Or use default (buck2)
let layout = default_layout();
}

Available Layouts

#![allow(unused)]
fn main() {
use composition::layout::available_layouts;

for name in available_layouts() {
    println!("Available: {}", name);
}
}

Built-in Layouts

Buck2Layout

The default layout for Buck2 projects:

• Maps cells to external/<cell>/
• Generates .buckconfig with cell mappings
• Generates .buckroot marker

#![allow(unused)]
fn main() {
use composition::layout::Buck2Layout;

let layout = Buck2Layout::new();
// or with custom prelude cell
let layout = Buck2Layout::with_prelude("custom-prelude");
}

BazelLayout

For Bazel-based projects:

• Maps cells to external/<cell>/
• Generates WORKSPACE file
• Generates root BUILD.bazel

#![allow(unused)]
fn main() {
use composition::layout::BazelLayout;

let layout = BazelLayout::new();
}

Testing Custom Layouts

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;
    use composition::layout::{LayoutContext, CellInfo};
    use std::path::PathBuf;

    fn test_context() -> LayoutContext {
        LayoutContext {
            mount_point: PathBuf::from("/firefly/test"),
            repo_root: PathBuf::from("/home/user/project"),
            source_dir_name: "root".to_string(),
            cell_prefix: "external".to_string(),
            cells: vec![
                CellInfo {
                    name: "godeps".to_string(),
                    source_path: PathBuf::from("/nix/store/xxx-godeps"),
                    editable: false,
                },
            ],
        }
    }

    #[test]
    fn test_map_dep() {
        let layout = PleaseLayout;
        let ctx = test_context();

        let mapped = layout.map_dep(&ctx, "godeps", Path::new("vendor/foo"));
        assert_eq!(
            mapped,
            Some(PathBuf::from("/firefly/test/third_party/go/vendor/foo"))
        );
    }

    #[test]
    fn test_generate_config() {
        let layout = PleaseLayout;
        let ctx = test_context();

        let configs = layout.generate_config(&ctx);
        assert_eq!(configs.len(), 1);
        assert_eq!(configs[0].path, PathBuf::from(".plzconfig"));
        assert!(configs[0].content.contains("[please]"));
    }

    #[test]
    fn test_supported_cells() {
        let layout = PleaseLayout;
        let ctx = test_context();

        let cells = layout.supported_cells(&ctx);
        assert!(cells.contains(&"godeps".to_string()));
    }
}
}

Best Practices

1. Keep map_dep simple - Just path manipulation, no I/O
2. Generate minimal configs - Only what the build system needs
3. Support all standard cells - godeps, rustdeps, pydeps, jsdeps
4. Use cell_path() helper - For consistent path construction
5. Test with real build systems - Verify generated configs work
6. Document cell expectations - What each cell should contain

API Reference

Module: composition::layout

Traits:

• Layout - Core layout trait

Structs:

• LayoutContext - Context for layout operations
• LayoutRegistry - Registry for custom layouts
• CellInfo - Information about a cell
• ConfigFile - Generated configuration file
• SimpleLayout - Quick layout without full trait impl
• Buck2Layout - Built-in Buck2 layout
• BazelLayout - Built-in Bazel layout

Functions:

• global_registry() - Get the global layout registry
• available_layouts() - List available layout names
• layout_by_name(name) - Get a layout by name
• default_layout() - Get the default layout (Buck2)

Type Aliases:

• BoxedLayout - Box<dyn Layout>
• LayoutFactory - fn() -> BoxedLayout

Dependency Generators

Tools that generate deps TOML files from native lock files.

Overview

Each language has a generator that:

1. Reads native lock files (go.sum, Cargo.lock, uv.lock)
2. Extracts dependency information
3. Prefetches packages to get Nix hashes
4. Outputs deps TOML for Nix cell building

Generator Structure

Input

Native lock file format (varies by language).

Output

TOML file with dependencies:

# go-deps.toml example
[deps]
[deps."github.com/pkg/errors"]
version = "v0.9.1"
hash = "sha256-xyz..."

[deps."golang.org/x/sys"]
version = "v0.15.0"
hash = "sha256-abc..."

What generators share

A generator's own code is its lock file parsing and its record type. The rest is the same for every generator:

• Flags: -o, --output; prefetching on by default, --no-prefetch to skip it; --no-cache to bypass turnkey's prefetch cache.
• Prefetching: the Nix SRI hash of a URL, of its unpacked contents for archives fetched with fetchzip, cached across runs and generators.
• Output: a header saying which generator wrote the file from what, and that tk sync regenerates it, then the record.

The Rust generators take all three from src/rust/deps-gen-kit: OutputArgs and PrefetchArgs to flatten into their clap Args, the Prefetcher seam (NixPrefetcher on the prefetch-cache crate, and MemoryPrefetcher for tests), and OutputArgs::write for a serde-serialized record. godeps-gen flattens the same flags, but prefetches through one nix-prefetch-cached --batch call (built on the same crate) and writes go-deps.toml line by line, in the layout its Go version wrote.

tk sync runs a generator from its language's sync rule (nix/buck2/languages.nix) and reads the deps file from stdout.

Existing Generators

godeps-gen (Go)

Located at src/cmd/godeps-gen/ (Rust).

godeps-gen -o go-deps.toml

Reads: go.work and its members' go.mod/go.sum, or go.mod, go.sum

rustdeps-gen (Rust)

Located at cmd/rustdeps-gen/.

rustdeps-gen --cargo-lock Cargo.lock \
  --platform linux-x86_64=x86_64-unknown-linux-gnu \
  --platform macos-arm64=aarch64-apple-darwin \
  -o rust-deps.toml

Reads: Cargo.lock and the workspace's manifests. Runs cargo tree once per --platform (<os>-<cpu>=<rust target triple>) and cargo metadata, both --locked, to record each crate's package slice: its features and normal dependencies as Cargo's feature resolver resolves them, each with the platforms it applies on (ADR 0006). It lists the workspace members' Cargo.toml files under manifests, which the Rust sync rule's target_sources makes sources of the file.

pydeps-gen (Python)

Located at cmd/pydeps-gen/.

pydeps-gen --lock pylock.toml -o python-deps.toml

Reads: pylock.toml, uv.lock, or requirements.txt

Records each distribution's pure (py3-none-any) wheel, hashed unpacked, and fails naming any distribution that has none (ADR 0013).

Go Dependency Handling

go.mod / go.work → go-deps.toml → one package per module + cell index (nix/lib/deps-cell) → tk materialize → .turnkey/godeps/

1. go.mod, or a root go.work and its members' go.mod files, define the build list; Go's own module resolution picks one version per module (ADR 0007)
2. go-deps.toml adds each module's Nix hash, and lists the go.work members (generated by godeps-gen)
3. Each module's own derivation (mkGoDepPackage in nix/lib/deps-cell/adapters/go.nix) applies the module's fixup and user patches, adds the assembly headers, and runs buckgen (src/cmd/buckgen/) on the module alone: one rules.star per Go package, labelled godeps//vendor/<import path>:<last component>. Two more outputs, targets and imports, list the Go packages rendered and the import paths they reference.
4. The cell index (godeps-index) lists each Go package's path, its module's store path and its subdirectory there, and a forwarding alias package for each referenced import path a go.work member owns (ADR 0008). tk materialize lays the cell out from it.

Rust Dependency Handling

Rust crates can have build.rs scripts that run during compilation. Since we can't run arbitrary code in Nix's sandbox, build script outputs must be handled manually.

The Standard Flow

Cargo.lock → rust-deps.toml → one package per crate + cell index (nix/lib/deps-cell) → tk materialize → .turnkey/rustdeps/

1. Cargo.lock defines exact versions and dependency graph
2. rust-deps.toml adds Nix hashes for each crate, and its package slice (generated by rustdeps-gen from cargo tree and cargo metadata)
3. Each crate's own derivation (mkRustCrates in nix/lib/deps-cell/adapters/rust.nix) applies the crate's fixup and user patches, and runs rust-rules-gen (src/cmd/rust-rules-gen/). That generates the crate's rules.star from the crate's directory, its slice, its own fixup, the platforms and the host, and nothing of any other crate: checks.rust-crate-isolation holds it to that. A second output, targets, lists the target names.
4. The cell index (rustdeps-index, built by mkCellIndex in nix/lib/deps-cell/default.nix) lists each package's path, store path and target names, the unversioned alias packages, the cell's .buckconfig and the SHA-256 of rust-deps.toml. tk materialize lays the cell out from it (ADR 0004).

rust-rules-gen is compiled because every crate's derivation starts it: on macOS, starting Python costs from about a second to about 40 s per process under parallel builds, which put a cold build of the per-crate derivations at roughly 15 minutes.

What Works Automatically

• Dependencies: As cargo resolves them, per platform (the package slice)
• Features: As cargo's feature resolver enables them, per platform
• Crate renaming: package = "real-name", and libraries named differently from their package
• Proc-macros: Detected from [lib] proc-macro = true
• Edition: Read from package.edition (defaults to 2015)
• Crate root: Detected from [lib] path or standard locations

What Requires a Fixup

Buck2 never runs a crate's build.rs. What a build script would produce comes from a fixup instead: a record for the crate in a fixup set, a module of class turnkeyFixups (ADR 0003). A repository brings fixup sets through turnkey.toolchains.buck2.fixups; the user manual's Dependency Fixups page covers bringing and publishing them. This section covers writing one and how turnkey applies it.

Every locked crate that has a build script needs a fixup whose buildScript either generates its output or says skip = true, even when its rustcFlags stand in for everything it does. Otherwise the cell fails to build, naming the crate. When one of turnkey's published families accounts for it, the error names the module to import.

Diagnosing Problems

Symptom: A crate has a build.rs, and no fixup says what stands in for it

The cell build fails with:

error: turnkey: zerocopy 0.8.37 has a build.rs, and no fixup says what stands in for it; give it one in turnkey.toolchains.buck2.fixups: ...

Diagnosis: read the crate's build.rs. If it only probes the rustc version or the target, or emits cfgs for features you don't use, the build needs nothing from it: rust.zerocopy.buildScript.skip = true. Otherwise, give it the fixup the symptoms below describe.

Symptom: Undefined cfg Flag

Error:

error[E0425]: cannot find value `fast_arithmetic` in this scope

Diagnosis: The crate's build.rs sets this via cargo:rustc-cfg=fast_arithmetic="64".

Solution:

rust.serde_json = {
  buildScript.skip = true; # the flag is all it produces
  rustcFlags = [ "--cfg" ''fast_arithmetic="64"'' ];
};

Symptom: Missing Generated File

Error:

error[E0432]: unresolved import `crate::private`

Diagnosis: The crate expects a file in OUT_DIR that build.rs generates.

Solution:

rust.serde.buildScript.generate = ctx: ''
  cat > "$OUT_DIR/private.rs" << 'EOF'
  #[doc(hidden)]
  pub mod __private${ctx.versionParts.patch} {
      pub use crate::private::*;
  }
  EOF
'';

A crate whose fixup generates output gets OUT_DIR = "out_dir" in its rules.

Symptom: Linker Error for Native Symbols

Error:

error: linking with `cc` failed: exit status: 1
  = note: undefined reference to `ring_core_0_17_14__OPENSSL_cpuid_setup'

Diagnosis: The crate has C/assembly code that build.rs compiles.

Solution: a build script that compiles it into $OUT_DIR, and the library it produces in nativeLibraries. See nix/fixups/rust/ring.nix and nix/fixups/rust/tree-sitter.nix.

The Fixup Record

A Rust fixup, as nix/lib/fixups/schema.nix declares it:

rust.my_crate = {
  # What stands in for build.rs: exactly one of generate or skip
  buildScript.generate = ctx: "...";   # or a string; or: buildScript.skip = true;

  rustcFlags = [ "--cfg" "my_flag" ];  # every platform
  env.MY_VAR = "value";                # the crate's compile environment
  patches = [ ./fix.patch ];           # -p1, relative to the crate's root
  nativeLibraries = [
    {
      name = ctx: "my_lib_${ctx.versionParts.patch}"; # a string, or a function of the context
      staticLib = "out_dir/libmy_lib.a";              # relative to the crate
      linkSearchPath = "out_dir";                     # the default
    }
  ];

  # Overlays: declarative additions per OS, per CPU, or per OS and CPU
  # pair, as select()s in the rules. A platform gets list fields from its
  # OS's overlay, then its CPU's, then its pair's. Two overlays giving one
  # platform an env variable different values fail evaluation. No build
  # script here: a crate has one, branching on ctx.platform.
  os.linux.rustcFlags = [ "--cfg" "linux_like" ];
  cpu.arm64.env.MY_ARCH = "arm64";
  platform."macos-arm64".rustcFlags = [ "--cfg" "apple_silicon" ];

  # Fields for the locked versions whose bounds hold; every match applies
  versions = [
    {
      when = { atLeast = "0.17"; below = "0.18"; };
      rustcFlags = [ "--cfg" "v017" ];
    }
  ];

  enable = true; # false drops a fixup an imported set brings
};

Every language's fixups have enable, patches, env and versions, keyed by the dependency's name in its own ecosystem (go."github.com/foo/bar", python.requests, …). Patches apply in every language; env is Rust-only for now, and an error elsewhere.

Merging is the module system's: lists concatenate in import order, and two sets giving one crate different build scripts, or one env variable different values, fail evaluation naming both files. Resolve it with lib.mkForce, disabledModules, or enable = false.

The Fixup Context

buildScript.generate and native library fields that are functions receive:

ctx = {
  name = "my_crate";
  version = "1.2.3-rc.1";
  versionParts = { major = "1"; minor = "2"; patch = "3"; pre = "rc.1"; };
  platform = { system = "aarch64-darwin"; os = "macos"; cpu = "arm64"; };
  pkgs = <nixpkgs>;
  lib = <nixpkgs lib>;
};

platform is the platform the cell is built on, which is the only one a native library the build script compiles exists for. turnkey links nativeLibraries on that platform only: building the crate for another platform fails in Buck2 ("no condition matched") rather than linking a missing library.

The build script runs in the crate's derivation, after its patches, with $CRATE_SRC its root and $OUT_DIR (created) its build script output directory.

Nix Interpolation vs Shell Escaping

In Nix multiline strings ('' ... ''):

rust.my_crate.buildScript.generate = ctx: ''
  # CORRECT: ${ctx.versionParts.patch} is Nix interpolation
  MY_VAR="${ctx.versionParts.patch}"

  # WRONG: ''${ctx.versionParts.patch} escapes the $ for the shell,
  # where it is undefined
  MY_VAR="''${ctx.versionParts.patch}"

  # CORRECT: $OUT_DIR is a shell variable
  echo "Output: $OUT_DIR"
'';

Rule: Use ${var} for Nix values, $var for shell variables.

turnkey's Own Fixups

turnkey's repository brings its own set, nix/fixups, one module per family: build-script-skips, fuser, nix, ring, rustix, serde, thiserror and tree-sitter. The flake publishes them as modules.turnkeyFixups.<family>, plus default importing them all. turnkey imports default itself; no other repository gets them unless it imports them.

Best Practices

1. Check build.rs first - Read the crate's build.rs to understand what it does
2. Start simple - skip, then rustcFlags, before a generating build script
3. Bound what is version-specific - Put it in a versions entry, so a new version is unaccounted for rather than silently wrong
4. Document complex fixups - Explain what the original build.rs does

Debugging Tips

Inspect Generated rules.star Files

cat .turnkey/rustdeps/vendor/serde_json@1.0.140/rules.star

Look for:

• rustc_flags - Should include cfg flags
• env - Should include OUT_DIR if the fixup generates build script output
• deps - Dependencies resolved correctly

Check Fixup Output

ls -la .turnkey/rustdeps/vendor/ring@0.17.14/out_dir/

Trace Feature Resolution

grep -A20 "rust_library" .turnkey/rustdeps/vendor/serde@*/rules.star | grep features

Creating a New Generator

1. Create CLI Tool

Create a CLI tool in cmd/newlang-gen/:

// cmd/newlang-gen/main.go
package main

import (
    "flag"
    "os"
    // ...
)

func main() {
    lockFile := flag.String("lock", "newlang.lock", "Path to lock file")
    output := flag.String("o", "", "Output file (default: stdout)")
    prefetch := flag.Bool("prefetch", true, "Fetch Nix hashes")
    flag.Parse()

    // 1. Parse lock file
    deps := parseLockFile(*lockFile)

    // 2. Prefetch packages if requested
    if *prefetch {
        prefetchHashes(deps)
    }

    // 3. Output TOML
    outputTOML(deps, *output)
}

2. Create Cell Builder

Add an adapter under nix/lib/deps-cell/adapters/, for example newlang.nix. A minimal cell builder looks like this:

{ pkgs, lib, depsFile }:

let
  deps = builtins.fromTOML (builtins.readFile depsFile);

  fetchDep = name: info:
    pkgs.fetchurl {
      url = info.url;
      hash = info.hash;
    };

  depSources = lib.mapAttrs fetchDep deps.deps;
in
pkgs.runCommand "newlang-deps-cell" {} ''
  mkdir -p $out

  # Generate cell .buckconfig
  cat > $out/.buckconfig << 'EOF'
  [cells]
      newlang-deps = .
  [buildfile]
      name = rules.star
  EOF

  # Generate rules.star for each dependency
  ${lib.concatStrings (lib.mapAttrsToList (name: src: ''
    mkdir -p $out/${name}
    cp -r ${src}/* $out/${name}/
    cat > $out/${name}/rules.star << 'EOF'
    # Generated build rules for ${name}
    newlang_library(
        name = "${lib.last (lib.splitString "/" name)}",
        srcs = glob(["*.newlang"]),
        visibility = ["PUBLIC"],
    )
    EOF
  '') depSources)}
''

3. Add the Language's Record

Add a record to nix/buck2/languages.nix: the cell name, depsFile (the deps file's default name), the generator package, mkCell (which calls the adapter) and syncRules. Everything else follows from the record: the flake-parts module builds the cell, and the devenv module adds the cell to .buckconfig, symlinks it under .turnkey/, puts the generator on the shell's PATH and writes the sync rules into .turnkey/sync.toml, with the cell and deps file in its [[languages]].

Rules sync (src/rust/rules-syncer, a library tk calls) reads [[languages]] and creates, for each language, the plug-in registered under the record's name in Mapper::new (src/rust/rules-syncer/src/mapper/mod.rs): a language without one is an error, so add the plug-in with the record, and update the checked-in src/rust/rules-syncer/testdata/sync.toml (checks.sync-config-contract prints the file to copy).

4. Add Configuration Options

Add the language's options to nix/buck2/options.nix: at least enable, cell and depsFile, plus whatever its sync rules read.

Testing

# Generate deps file
newlang-gen > newlang-deps.toml

# Verify it's valid TOML
nix eval --expr 'builtins.fromTOML (builtins.readFile ./newlang-deps.toml)'

# Build the cell
nix build .#newlang-deps-cell

# Check generated content
ls result/

Development Setup

Set up your environment to contribute to Turnkey.

Prerequisites

• Nix with flakes enabled
• direnv (recommended)
• Git

Clone and Enter Shell

git clone https://github.com/firefly-engineering/turnkey.git
cd turnkey
direnv allow  # or: nix develop

Repository Layout

turnkey/
├── flake.nix           # Main flake (self-usage example)
├── toolchain.toml      # Example toolchain config
├── nix/
│   ├── flake-parts/    # Flake-parts module
│   ├── devenv/         # Devenv module
│   ├── registry/       # Default registry
│   ├── buck2/          # Buck2 integration
│   └── packages/       # Tool packages
├── cmd/                # CLI tools (Go)
├── docs/               # Documentation
└── examples/           # Example projects

Making Changes

Nix Code

1. Edit files in nix/
2. Stage changes: git add nix/
3. Re-enter shell to test: exit && nix develop

Go Code

1. Edit files in cmd/
2. Build: tk build //src/cmd/...
3. Run: tk run //src/cmd/mytool:mytool

Documentation

1. Edit files in docs/
2. Build book: tk build //docs/user-manual:user-manual
3. Preview: tk run //docs/user-manual:user-manual

Running Tests

# All tests
tk test //...

# Specific package
tk test //src/rust/project-sync:project-sync-test

Pre-commit Hooks

Turnkey uses pre-commit hooks for:

• Nix flake check
• Monorepo dependency check
• Rust edition check

Hooks run automatically on commit.

Code Style

Nix

Formatting

• 2 space indentation
• Multi-line function parameters
• Aligned braces

{
  config,
  pkgs,
  lib,
  ...
}:

Module Pattern

{
  options = {
    # Option definitions
  };

  config = lib.mkIf cfg.enable {
    # Implementation
  };
}

Naming

• Use descriptive attribute names
• camelCase for local variables
• kebab-case for package names

Starlark (Buck2)

Rule Definitions

def _my_rule_impl(ctx: AnalysisContext) -> list[Provider]:
    """Implementation of my_rule.

    Args:
        ctx: Analysis context from Buck2
    """
    pass

my_rule = rule(
    impl = _my_rule_impl,
    attrs = {
        "srcs": attrs.list(attrs.source()),
    },
    doc = "Short description of the rule.",
)

Naming

• snake_case for functions and variables
• PascalCase for providers
• _prefix for private functions

Go

Standard Go formatting with gofmt.

Package Comments

// Package syncer provides dependency synchronization.
package syncer

Commit Messages

Use conventional commits:

feat: add zig toolchain support
fix: correct Python deps cell generation
docs: update troubleshooting guide
refactor: simplify registry pattern

Prefix types:

• feat: - New features
• fix: - Bug fixes
• docs: - Documentation
• refactor: - Code restructuring
• test: - Test additions
• chore: - Maintenance

Testing

Running Tests

All Tests

tk test //...

Specific Packages

# Rust crates
tk test //src/rust/project-sync:project-sync-test
tk test //src/rust/prefetch-cache:prefetch-cache-test

# Python modules
tk test //src/python/cargo:test_features

Test Categories

Unit Tests

Located alongside source code:

pkg/
├── syncer.go
└── syncer_test.go

Integration Tests

Located in e2e/:

e2e/
├── fixtures/
│   ├── greenfield-go/
│   └── multi-language/
└── run_e2e.sh

Nix Testing

Flake Check

nix flake check

Validates:

• Module definitions
• Package builds
• Template validity

Derivation Builds

# Build specific package
nix build .#godeps-gen

# Build prelude
nix build .#turnkey-prelude

Manual Testing

New Toolchain

1. Add to registry
2. Add to mappings
3. Add to toolchain.toml
4. Enter shell
5. Verify tk targets toolchains//...

Prelude Extension

1. Create extension files
2. Stage: git add nix/buck2/prelude-extensions/
3. Rebuild: nix build .#turnkey-prelude
4. Verify files in output

Dependency Cell

1. Generate deps file
2. Build cell: nix build .#godeps-cell (example)
3. Verify rules.star content

CI

Pre-commit hooks run:

1. nix flake check
2. monorepo-dep-check
3. rust-edition-check

All hooks must pass before commit.

Submitting Changes

Before You Start

1. Check existing issues for related work
2. For large changes, open an issue first to discuss approach
3. Fork the repository

Development Workflow

1. Create a feature branch:

   git checkout -b feat/my-feature
   

2. Make your changes

3. Test thoroughly:

   tk test //...
   nix flake check
   

4. Commit with conventional message:

   git commit -m "feat: add zig toolchain support"
   

Pull Request Process

1. Push your branch:

   git push -u origin feat/my-feature
   

2. Open a Pull Request on GitHub

3. Fill out the PR template:

   • Summary of changes
   • Test plan
   • Related issues

4. Wait for review

PR Checklist

• Tests pass (tk test //...)
• Nix flake checks pass (nix flake check)
• Pre-commit hooks pass
• Documentation updated if needed
• Commit messages follow convention

Review Process

• Maintainers will review within a few days
• Address feedback with additional commits
• Once approved, maintainer will merge

After Merge

• Delete your feature branch
• Pull latest main
• Thanks for contributing!

Getting Help

• Open an issue for questions
• Tag maintainers if stuck on review
• Check existing PRs for examples

Bumping the Pinned buck2 Release

Each turnkey revision ships exactly one buck2 release (ADR 0002): the binary and the upstream prelude built with it, both from toolbox, and the buck2 source revision they come from. They are named together in one record, in nix/buck2/buck2-source.nix. A bump moves all of them in one change, and nothing lands until the parity suite passes against the new release.

This checklist records what the move to buck2 2026-09-15 took.

1. Update toolbox

Toolbox has to carry the new release first, under its date, for both buck2 and buck2-prelude:

nix flake update toolbox
nix eval --quiet --json --impure --expr '
  let r = (builtins.getFlake (toString ./.)).lib.defaultTellerRegistry "aarch64-darwin";
  in { buck2 = builtins.attrNames r.buck2.versions;
       prelude = builtins.attrNames r.buck2-prelude.versions; }'

If the date is missing from either list, add it to toolbox first. Don't work around it in turnkey.

A toolbox update moves every other toolchain as well. Expect fallout that has nothing to do with buck2. Last time:

• a meta-package changed shape: typescript's default became 7, which has no tsc.js, so the typescript mapping had to take node and tsc from the declared meta-package;
• a pinned version disappeared: nix 2.34.1 was dropped from toolbox;
• devenv moved with toolbox and brought an import-from-derivation, which broke nix flake check --no-build on a clean store. The fix was to take task.package from the devenv flake's devenv-tasks.

2. Fill in the pinned record

In nix/buck2/buck2-source.nix, set every field of pinned for the new release:

If toolbox's prelude for that date is a different commit than preludeRev, evaluation fails and names both commits. Fix the toolbox entry.

3. Port the prelude patches

Follow nix/patches/prelude/README.md: dry-run each patch against the new upstream prelude, and redo in place the ones that no longer apply. There is one patch set, for the pinned release only; nix build .#turnkey-prelude fails if any patch doesn't apply, so a forgotten port can't ship unpatched.

Read upstream prelude changes that touch the patched code, not just the hunks that fail to apply. In 2026-07-01 the new preludes gave tests without a remote-execution profile a non-caching local executor, which the caching helper (nix/buck2/prelude-extensions/test_caching/test_caching.bzl) now replaces. A change like that applies cleanly and only shows up in what the rules hand buck2, so check it:

python3 src/cmd/check-test-caching/__main__.py

It analyses a test target of every cache-safe rule with test result caching on and off, and must report that every target matches.

4. Check the protocol and the parity suite

The test runner's protocol code is regenerated from the new rev. Protocol changes show up as build failures in turnkey-test-runner. Changes to the event log the parity suite reads show up as problems in the suite's output. In 2026-09-15 buck2 renamed the TestRun span, so the suite found no action digests and reported that the requests weren't compared.

In a fresh shell (nix develop --impure):

buck2 --version     # <version>-<rev>, matching the pinned record
python3 src/cmd/check-test-runner-parity/__main__.py

Every scenario must report matches. The last line is the summary, for example:

parity: 3/3 scenarios match on 30 targets (buck2 2026-09-14-6507dd157a6f81a810c48583edf1758dd0c337c5, arm64-darwin)

CI runs the same suite on Linux for a pull request that changes the pin, the test runner or flake.lock (.github/workflows/test-runner-parity.yaml), in the slim ci shell (nix develop .#ci --impure, from .github/toolchain.toml). It doesn't replace the run above: CI covers x86_64-linux only, and the commit carries your summary line.

5. Check test result caching end to end

tk build //...
tk test //...        # every test runs; passes are recorded
tk test //...        # every test is reported as recorded (reused without running)

Check that recorded results also work from a second checkout of the same commit.

6. Check the daemon records turnkey-composed reads

After a FUSE mount, turnkey-composed kills the buck2 daemons of the projects inside the mount point. It finds them from buck2's own records, not through buck2 (src/rust/composition/src/buckd.rs): ~/.buck/buckd/<project root>/<isolation dir>/buckd.info, a JSON file holding the daemon's pid, and the daemon's command line, --isolation-dir <dir> daemon. Check that the new release still writes them so: InvocationPaths::daemon_dir in buck2's app/buck2_common/src/invocation_paths.rs, and with a daemon running,

cat ~/.buck/buckd$PWD/v2/buckd.info
ps -p <pid> -o args=

7. Run the CI gates locally

Run what .github/workflows/ci.yaml, docs.yaml and cachix.yaml run:

nix flake check --no-build --impure
nix build --no-link .#turnkey-prelude .#tk .#godeps-gen .#rustdeps-gen .#pydeps-gen .#jsdeps-gen
nix develop .#docs --impure -c bash -c 'for b in landing user-manual developer-manual; do mdbook build docs/$b --dest-dir $(mktemp -d); done'
nix build --no-link --impure --expr '
  let f = builtins.getFlake (toString ./.); p = f.packages.aarch64-darwin;
  in map (n: p.${n}) f.lib.publicPackages'

8. Commit

Put the whole bump in one commit: the toolbox update, the pinned record, the patch set and any fallout. Paste the parity summary line into the commit message, so the log shows the gate ran, against which release, and on which platform.