Introduction

This manual is for developers who want to understand, extend, or contribute to Turnkey.

What You'll Learn

• Turnkey's architecture and design principles
• How the Nix module system works
• Adding new toolchains and language support
• Creating custom Buck2 rules
• Contributing to the project

Prerequisites

You should be familiar with:

• Nix - Flakes, derivations, module system
• Buck2 - Targets, rules, toolchains
• Starlark - Buck2's configuration language

Project Philosophy

Turnkey follows these principles:

1. Simplicity over features - Solve common cases elegantly
2. Declarative configuration - TOML in, working environment out
3. Reproducibility - Same inputs = same outputs, always
4. Composition - Build complex systems from simple parts
5. Transparency - Generated code should be readable

Repository Structure

turnkey/
├── nix/
│   ├── flake-parts/turnkey/  # Flake-parts integration
│   ├── devenv/turnkey/       # Devenv shell configuration
│   ├── registry/             # Default toolchain registry
│   ├── buck2/                # Buck2 cell generation
│   │   ├── prelude.nix       # Prelude derivation
│   │   ├── mappings.nix      # Toolchain mappings
│   │   └── prelude-extensions/
│   ├── packages/             # Tool packages
│   └── patches/              # Upstream patches
├── cmd/                      # CLI tools (Go)
├── docs/                     # Documentation
└── examples/                 # Example projects

Architecture Overview

Turnkey uses a layered architecture to transform simple TOML declarations into working development environments.

Data Flow

toolchain.toml
    ↓
Flake-parts module (perSystem options)
    ↓
Devenv module (shell configuration)
    ↓
Registry (name → package resolution)
    ↓
Buck2 cell generation (toolchains, deps)
    ↓
Working development shell

Layer Responsibilities

toolchain.toml

Simple declaration of needed tools:

[toolchains]
go = {}
rust = {}

Flake-Parts Module

• Exposes turnkey.toolchains options at perSystem level
• Builds dependency cells from deps files
• Creates devenv shell configurations
• Located at nix/flake-parts/turnkey/default.nix

Devenv Module

• Receives registry and declaration file
• Resolves toolchain names to packages
• Adds packages to shell PATH
• Generates Buck2 cells on shell entry
• Located at nix/devenv/turnkey/default.nix

Registry

• Maps toolchain names to Nix packages
• Versioned format: { go = { versions = { ... }; default = "..."; }; }
• Extensible by users via registryExtensions or custom overlays
• Default registry provided by teller

Buck2 Cells

• Generated at shell entry time
• Toolchains cell with language-specific rules
• Dependency cells (godeps, rustdeps, pydeps)
• Symlinked into .turnkey/

Key Design Decisions

1. Nix for package resolution - Leverages nixpkgs for reproducibility
2. Devenv for shell management - Proven shell environment tooling
3. Generated Buck2 cells - Dynamic, not committed to repo
4. Prelude from Nix - Pinned, patched, extended prelude

Module System

Turnkey uses NixOS-style modules for configuration.

Module Layers

Flake-Parts Module

Located at nix/flake-parts/turnkey/default.nix.

Provides perSystem options:

options.perSystem = mkPerSystemOption ({...}: {
  options.turnkey.toolchains = {
    enable = mkOption { type = types.bool; default = true; };
    declarationFiles = mkOption { type = types.attrsOf types.path; };
    registry = mkOption { type = types.lazyAttrsOf types.package; };
    buck2 = {
      enable = mkOption { ... };
      go.depsFile = mkOption { ... };
      rust.depsFile = mkOption { ... };
      # ...
    };
  };
});

Devenv Module

Located at nix/devenv/turnkey/default.nix.

Receives configuration from flake-parts:

options.turnkey = {
  enable = mkOption { type = types.bool; };
  declarationFile = mkOption { type = types.path; };
  registry = mkOption { type = types.lazyAttrsOf types.package; };
};

config = lib.mkIf cfg.enable {
  packages = resolvedPackages;
};

Configuration Flow

1. User sets turnkey.toolchains in their flake
2. Flake-parts module creates shell configs
3. Each shell config imports devenv module
4. Devenv module resolves packages from registry

Extending Options

Add new options in the appropriate module:

# In flake-parts module for user-facing API
options.turnkey.toolchains.myFeature = mkOption {
  type = types.bool;
  default = false;
  description = "Enable my feature";
};

# Pass to devenv module in mkShellConfig
turnkey.myFeature = cfg.myFeature;

Registry Pattern

The registry maps toolchain names to versioned Nix packages. The core registry library and default registry live in teller, a standalone Nix flake that turnkey depends on.

Structure

The default registry is provided by teller (registry/default.nix in the teller repo):

{ pkgs, lib ? pkgs.lib }:

let
  # Helper for single-version entries
  single = pkg: {
    versions = { "default" = pkg; };
    default = "default";
  };
in {
  go = single pkgs.go;
  rust = single pkgs.rustc;
  python = single pkgs.python3;
  # ...
}

Versioned Format

Each registry entry has:

<toolchain-name> = {
  versions = {
    "<version-string>" = <derivation>;
    # ...
  };
  default = "<version-string>";  # Must match a key in versions
};

For example, a multi-version entry:

go = {
  versions = {
    "1.21" = pkgs.go_1_21;
    "1.22" = pkgs.go_1_22;
    "1.23" = pkgs.go_1_23;
  };
  default = "1.23";
};

Design Principles

1. Versioned - Each toolchain can have multiple versions
2. Lazy evaluation - Only builds what's used
3. Composable - Multiple registries can be merged via overlays
4. User-overridable - Versions and defaults can be customized

Library Functions

Teller provides helpers in teller.lib (system-independent):

resolveTool

Resolves a toolchain from the registry:

# Usage
go = teller.lib.resolveTool registry "go" {};           # Use default
go122 = teller.lib.resolveTool registry "go" { version = "1.22"; };

resolveToolchains

Resolves all toolchains from a parsed toolchain.toml:

declaration = builtins.fromTOML (builtins.readFile ./toolchain.toml);
packages = teller.lib.resolveToolchains registry declaration;

mkRegistryOverlay

Creates overlays with two-level merging for registry composition:

overlays.default = teller.lib.mkRegistryOverlay (final: prev: {
  go = {
    versions = { "1.24" = final.go_1_24; };
    default = "1.24";
  };
});

When composed, versions are merged additively and default is overridden.

mkMetaPackage

Bundles multiple tools into a single derivation:

rust = {
  versions = {
    "1.80" = teller.lib.mkMetaPackage {
      inherit pkgs;
      name = "rust-1.80";
      components = {
        rustc = final.rustc;
        cargo = final.cargo;
        clippy = final.clippy;
        rustfmt = final.rustfmt;
      };
    };
  };
  default = "1.80";
};

Default Sourcing

The turnkey.toolchains flake-parts module options tellerLib and tellerRegistry default to the teller + toolbox setup that turnkey bundles, so consumers don't have to wire them up unless they need a private registry or an alternate teller revision.

The defaults are also exposed as standalone helpers on turnkey's lib output, reachable from any downstream flake:

# inputs.turnkey.lib.defaultTellerLib       : teller.lib (system-agnostic)
# inputs.turnkey.lib.defaultTellerRegistry  : system → registry attrset

So a typical consumer flake reduces to:

turnkey.toolchains = {
  enable = true;
  declarationFiles.default = ./toolchain.toml;
  buck2 = { ... };
};

Override scenarios:

• Private registry overlay — extend the toolbox-backed default with extra overlays:

  turnkey.toolchains.tellerRegistry =
    (import inputs.nixpkgs {
      inherit system;
      overlays = [
        inputs.teller.overlays.default
        inputs.toolbox.overlays.default
        inputs.my-org-overlay.overlays.default
      ];
    }).turnkeyRegistry;
  

• Alternate teller revision — pin a fork or a specific teller commit:

  turnkey.toolchains.tellerLib = inputs.my-teller-fork.lib;
  

Or reach the defaults directly for ad-hoc composition outside the module:

let
  registry = inputs.turnkey.lib.defaultTellerRegistry system;
  resolvedBuck2 = inputs.turnkey.lib.defaultTellerLib.resolveTool registry "buck2" {};
in ...

How It's Used

The flake-parts module injects tellerLib into the devenv module:

# In the devenv module:
turnkeyLib = cfg.tellerLib;

# Parse toolchain.toml and resolve all toolchains
declaration = builtins.fromTOML (builtins.readFile cfg.declarationFile);
packages = turnkeyLib.resolveToolchains cfg.registry declaration;

Extending the Registry

Via registryExtensions

In your flake.nix:

turnkey.toolchains = {
  registryExtensions = let
    single = pkg: { versions = { "default" = pkg; }; default = "default"; };
  in {
    mytool = single myCustomPackage;
    # Add versions to existing toolchain
    go = {
      versions = { "1.24" = pkgs.go_1_24; };
      default = "1.24";  # Override default
    };
  };
};

Via Custom Registry Overlay

For reusable registries, create a flake that exports an overlay:

# my-registry/flake.nix
{
  inputs.teller.url = "github:firefly-engineering/teller";

  outputs = { teller, ... }: {
    overlays.default = teller.lib.mkRegistryOverlay (final: prev: {
      zig = {
        versions = {
          "0.11" = final.zig_0_11;
          "0.12" = final.zig_0_12;
        };
        default = "0.12";
      };
    });
  };
}

Consumers compose overlays:

pkgs = import nixpkgs {
  overlays = [
    official-registry.overlays.default
    my-registry.overlays.default  # Versions merge!
  ];
};

Internal Tools

Dependency generators (godeps-gen, rustdeps-gen, etc.) are not in the registry. They're internal turnkey tools that are automatically included when the corresponding language is enabled.

Adding to Default Registry

To add a new standard toolchain, contribute to teller's registry/default.nix.

To add a turnkey-specific tool, add it to registryExtensions in turnkey's flake.nix.

# Single version (most common)
zig = single pkgs.zig;

# Multiple versions
nodejs = {
  versions = {
    "18" = pkgs.nodejs_18;
    "20" = pkgs.nodejs_20;
    "22" = pkgs.nodejs_22;
  };
  default = "20";
};

Buck2 Integration

This document describes how Turnkey integrates with Buck2 and covers the architecture of cell generation.

Overview

Turnkey generates Buck2 cells at shell entry time. Cells are directories that Buck2 treats as separate projects with their own configuration and build rules.

Cell Resolution

Buck2 cell resolution is entirely configuration-driven through .buckconfig files. There is no environment variable like CELL_PATH for overriding cell locations at runtime.

Primary Configuration: [cells] Section

Cells are defined in .buckconfig files:

[cells]
    root = .
    prelude = path/to/prelude
    toolchains = path/to/toolchains

Key points:

• Paths are relative to the directory containing the .buckconfig file
• Left side: cell alias (alphanumeric + underscores only)
• Right side: filesystem path

Configuration File Precedence

Buck2 reads configuration from multiple sources (highest to lowest precedence):

1. Command-line: --config, --config-file, --flagfile
2. .buckconfig.local (repo root)
3. .buckconfig (repo root)
4. .buckconfig.d/ folder (repo root)
5. ~/.buckconfig.local (user home)
6. ~/.buckconfig.d/ (user home)
7. /etc/buckconfig (global)
8. /etc/buckconfig.d/ (global)

Reference: app/buck2_common/src/legacy_configs/path.rs:35-60

Cell Override Restrictions

Buck2 explicitly bans overriding cell definitions using the --config command-line flag:

#![allow(unused)]
fn main() {
// app/buck2_common/src/legacy_configs/parser.rs:133-144
for banned_section in ["repositories", "cells"] {
    if config_pair.section == banned_section {
        return Err(
            ConfigArgumentParseError::CellOverrideViaCliConfig(banned_section).into(),
        );
    };
}
}

This means:

• buck2 --config cells.prelude=/path → ERROR
• buck2 --config repositories.toolchains=/path → ERROR

Solution: Use --config-file instead, which has no restrictions on [cells] sections.

Generated Cells

Toolchains Cell (.turnkey/toolchains/)

Contains toolchain rules for each declared language:

# Generated rules.star
load("@prelude//toolchains/go:system_go_toolchain.bzl", "system_go_toolchain")

system_go_toolchain(
    name = "go",
    visibility = ["PUBLIC"],
)

Prelude Cell (.turnkey/prelude/)

Symlink to Nix-built prelude with:

• Upstream buck2-prelude at pinned commit
• Applied patches
• Custom extensions (TypeScript, mdbook, etc.)

Dependency Cells

• godeps/ - Go third-party packages
• rustdeps/ - Rust crates
• pydeps/ - Python packages

Toolchain Mappings

Located at nix/buck2/mappings.nix:

{
  go = {
    skip = false;
    targets = [{
      name = "go";
      rule = "system_go_toolchain";
      load = "@prelude//toolchains/go:system_go_toolchain.bzl";
      visibility = [ "PUBLIC" ];
      dynamicAttrs = registry: {
        go_binary = "${registry.go}/bin/go";
      };
    }];
    implicitDependencies = [ "python" "cxx" ];
    runtimeDependencies = [ ];
  };
}

Mapping Structure

Generation Process

1. Devenv shell entry hook runs
2. nix/devenv/turnkey/buck2.nix generates toolchains cell
3. Dependency cells built from deps files
4. Symlinks created in .turnkey/

Nix Integration Strategy

Turnkey uses symlinks to Nix store paths for Buck2 cells:

.turnkey/
├── toolchains -> /nix/store/...-turnkey-toolchains-cell
├── godeps     -> /nix/store/...-go-deps-cell
├── rustdeps   -> /nix/store/...-rust-deps-cell
├── pydeps     -> /nix/store/...-python-deps-cell
└── prelude    -> /nix/store/...-turnkey-prelude

Config File Solution

Since --config can't override cells but --config-file can, Turnkey generates a complete .buckconfig in the Nix store and symlinks to it:

# Generated .buckconfig in Nix store
[cells]
    root = .
    prelude = /nix/store/xxx-prelude
    toolchains = /nix/store/yyy-toolchains
    godeps = /nix/store/zzz-godeps

This allows each Nix environment to provide its own prelude/toolchains while sharing the same Buck2 project configuration.

Benefits

• Multiple shells from same git checkout
• Each shell can point to different toolchains/prelude
• No modification of version-controlled files
• Clean integration with Nix's environment management

Config File Capabilities

Value Interpolation

Reference other config values:

[custom]
    base_path = /nix/store/abc123

[cells]
    prelude = $(config custom.base_path)/prelude
    toolchains = $(config custom.base_path)/toolchains

Reference: app/buck2_common/src/legacy_configs/parser/resolver.rs:150-153

File Includes

Include other configuration files:

# Required include
<file:path/to/other.buckconfig>

# Optional include (no error if missing)
<?file:path/to/optional.buckconfig>

Limitations

No Environment Variable Substitution: Buck2 does not support $(env VAR_NAME) syntax in config files. Only $(config section.key) is supported.

This is why the generated config file approach is necessary.

External Cells

Buck2 supports external cells (bundled or git-based):

Bundled External Cells

[cells]
    prelude = prelude/

[external_cells]
    prelude = bundled

Git External Cells

[cells]
    prelude = prelude/

[external_cells]
    prelude = git

[external_cell_prelude]
    git_origin = https://github.com/facebook/buck2-prelude.git
    commit_hash = <40-char-sha1-hash>

Note: External cells don't solve the dynamic path problem since they still require configuration file changes.

Prelude Customization

The prelude is built by nix/buck2/prelude.nix:

1. Fetch upstream buck2-prelude
2. Apply patches from nix/patches/prelude/
3. Copy extensions from nix/buck2/prelude-extensions/

See Prelude Extensions for adding custom rules.

Key Source Files (Buck2)

Key Source Files (Turnkey)

Build System Abstraction

Turnkey supports multiple build systems through abstraction layers that separate build-system-agnostic specifications from build-system-specific rule generation.

Overview

The abstraction pattern allows:

1. Single source of truth - Dependency specifications remain build-system-agnostic
2. Pluggable generators - Each build system implements its own rule generation
3. Easy extensibility - Adding new build systems requires only implementing generator protocols

┌─────────────────────────────────────────────────────────────────┐
│                   Generic Specification                         │
│  (NativeLibrarySpec, CellInfo, etc.)                            │
└─────────────────────────────────────────────────────────────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
     ┌────────────┐   ┌────────────┐   ┌────────────┐
     │   Buck2    │   │   Bazel    │   │   Future   │
     │ Generator  │   │ Generator  │   │ Generators │
     └────────────┘   └────────────┘   └────────────┘
              │               │               │
              ▼               ▼               ▼
     ┌────────────┐   ┌────────────┐   ┌────────────┐
     │ rules.star │   │ BUILD.bazel│   │    ...     │
     │ .buckconfig│   │ WORKSPACE  │   │            │
     └────────────┘   └────────────┘   └────────────┘

Native Library Generation

The Problem

Pre-compiled native libraries (like ring's crypto code) need different rules for each build system:

The Abstraction

NativeLibrarySpec

A build-system-agnostic specification for native libraries:

@dataclass
class NativeLibrarySpec:
    """Build-system-agnostic specification for a native library."""

    lib_name: str           # Target name (e.g., "ring_core_0_17_14__")
    static_lib_path: str    # Path to .a file (e.g., "out_dir/libring_core.a")
    link_search_path: str   # Rustc -L path (default: "out_dir")

This contains only the information needed to describe the library, not how to build it.

NativeLibraryGenerator Protocol

Build systems implement this protocol:

class NativeLibraryGenerator(Protocol):
    """Protocol for generating native library rules."""

    def generate(self, spec: NativeLibrarySpec) -> GeneratedRules:
        """Generate build rules for a native library."""
        ...

    @property
    def name(self) -> str:
        """The build system name (e.g., 'buck2', 'bazel')."""
        ...

GeneratedRules

The output from a generator:

@dataclass
class GeneratedRules:
    """Result of generating native library rules."""

    rules_content: str          # Generated rule definitions
    rules_to_load: list[str]    # Rules to load (e.g., ["prebuilt_cxx_library"])
    extra_deps: list[str]       # Dependencies to add to the crate
    extra_rustc_flags: list[str] # Rustc flags for linking

Implementation Examples

Buck2 Generator

class Buck2NativeLibraryGenerator:
    @property
    def name(self) -> str:
        return "buck2"

    def generate(self, spec: NativeLibrarySpec) -> GeneratedRules:
        lines = [
            "export_file(",
            f'    name = "{spec.lib_name}_file",',
            f'    src = "{spec.static_lib_path}",',
            '    visibility = ["PUBLIC"],',
            ")",
            "",
            "prebuilt_cxx_library(",
            f'    name = "{spec.lib_name}",',
            f'    static_lib = ":{spec.lib_name}_file",',
            "    link_whole = True,",
            '    preferred_linkage = "static",',
            '    visibility = ["PUBLIC"],',
            ")",
        ]

        return GeneratedRules(
            rules_content="\n".join(lines),
            rules_to_load=["prebuilt_cxx_library", "export_file"],
            extra_deps=[f":{spec.lib_name}"],
            extra_rustc_flags=[f"-Lnative={spec.link_search_path}"],
        )

Generated output:

export_file(
    name = "ring_core_0_17_14___file",
    src = "out_dir/libring_core_0_17_14__.a",
    visibility = ["PUBLIC"],
)

prebuilt_cxx_library(
    name = "ring_core_0_17_14__",
    static_lib = ":ring_core_0_17_14___file",
    link_whole = True,
    preferred_linkage = "static",
    visibility = ["PUBLIC"],
)

Bazel Generator

class BazelNativeLibraryGenerator:
    @property
    def name(self) -> str:
        return "bazel"

    def generate(self, spec: NativeLibrarySpec) -> GeneratedRules:
        lines = [
            "cc_import(",
            f'    name = "{spec.lib_name}",',
            f'    static_library = "{spec.static_lib_path}",',
            '    visibility = ["//visibility:public"],',
            ")",
        ]

        return GeneratedRules(
            rules_content="\n".join(lines),
            rules_to_load=["cc_import"],
            extra_deps=[f":{spec.lib_name}"],
            extra_rustc_flags=[f"-Lnative={spec.link_search_path}"],
        )

Generated output:

cc_import(
    name = "ring_core_0_17_14__",
    static_library = "out_dir/libring_core_0_17_14__.a",
    visibility = ["//visibility:public"],
)

File Locations

Usage in Generator

The generator.py uses the abstraction:

from python.buildsystem.native_library import NativeLibrarySpec
from python.buildsystem.buck2 import buck2_generator

def generate_buck_file(..., native_lib_info: dict | None = None) -> str:
    if native_lib_info:
        spec = NativeLibrarySpec.from_dict(native_lib_info)
        generated = buck2_generator.generate(spec)

        rules_to_load.extend(generated.rules_to_load)
        deps = deps + generated.extra_deps
        rustc_flags = rustc_flags + generated.extra_rustc_flags

Layout Trait (FUSE Composition)

The composition layer uses a similar pattern for file system layouts.

Layout Trait

#![allow(unused)]
fn main() {
pub trait Layout: Send + Sync {
    /// Get the layout name (e.g., "buck2", "bazel")
    fn name(&self) -> &'static str;

    /// Map a dependency path to its location in the composed view
    fn map_dep(&self, ctx: &LayoutContext, cell: &str, path: &Path) -> Option<PathBuf>;

    /// Generate configuration files for this build system
    fn generate_config(&self, ctx: &LayoutContext) -> Vec<ConfigFile>;

    /// Get the list of cells this layout supports
    fn supported_cells(&self, ctx: &LayoutContext) -> Vec<String>;
}
}

Buck2Layout Implementation

#![allow(unused)]
fn main() {
impl Layout for Buck2Layout {
    fn name(&self) -> &'static str { "buck2" }

    fn generate_config(&self, ctx: &LayoutContext) -> Vec<ConfigFile> {
        vec![
            self.generate_buckconfig(ctx),
            ConfigFile::new(".buckroot", "# Buck2 repository root marker\n"),
        ]
    }

    // ...
}
}

See src/rust/composition/src/layout.rs for the full implementation.

Adding a New Build System

1. Create Native Library Generator

# src/python/buildsystem/newbuild.py
from .native_library import NativeLibrarySpec, GeneratedRules

class NewBuildNativeLibraryGenerator:
    @property
    def name(self) -> str:
        return "newbuild"

    def generate(self, spec: NativeLibrarySpec) -> GeneratedRules:
        # Generate rules for your build system
        lines = [
            f'native_lib(name = "{spec.lib_name}", ...)',
        ]
        return GeneratedRules(
            rules_content="\n".join(lines),
            rules_to_load=["native_lib"],
            extra_deps=[f":{spec.lib_name}"],
            extra_rustc_flags=[f"-Lnative={spec.link_search_path}"],
        )

newbuild_generator = NewBuildNativeLibraryGenerator()

2. Create Layout Implementation (for FUSE)

#![allow(unused)]
fn main() {
// src/rust/composition/src/layouts/newbuild.rs
pub struct NewBuildLayout;

impl Layout for NewBuildLayout {
    fn name(&self) -> &'static str { "newbuild" }

    fn generate_config(&self, ctx: &LayoutContext) -> Vec<ConfigFile> {
        // Generate config files for your build system
        vec![ConfigFile::new("BUILD.newbuild", "# config")]
    }

    // ...
}
}

3. Register the Layout

#![allow(unused)]
fn main() {
// src/rust/composition/src/layout.rs
pub fn layout_by_name(name: &str) -> Option<BoxedLayout> {
    match name {
        "buck2" => Some(Box::new(Buck2Layout::new())),
        "newbuild" => Some(Box::new(NewBuildLayout::new())),
        _ => None,
    }
}
}

Design Principles

1. Specification vs Generation - Keep specifications generic, push build-system details to generators
2. Protocol-based - Use protocols/traits for loose coupling
3. Singleton instances - Generators are stateless, use module-level instances
4. Incremental adoption - New build systems can be added without changing existing code

FUSE Composition Layer

The FUSE composition layer provides a unified filesystem view of repositories and their dependencies. This document covers the architecture for developers extending or maintaining the composition system.

Architecture Overview

┌────────────────────────────────────────────────────────────────┐
│                   CompositionBackend trait                     │
├────────────────────────────────────────────────────────────────┤
│  ┌─────────────────────┐       ┌─────────────────────┐         │
│  │   FUSE Backend      │       │   Symlink Backend   │         │
│  │   (Development)     │       │   (CI / Fallback)   │         │
│  └─────────────────────┘       └─────────────────────┘         │
│              │                           │                     │
│              └───────────┬───────────────┘                     │
│                          ▼                                     │
│              ┌───────────────────────┐                         │
│              │   Composition API     │                         │
│              │   (shared interface)  │                         │
│              └───────────────────────┘                         │
└────────────────────────────────────────────────────────────────┘

Core Components

Backend Trait

The CompositionBackend trait (src/rust/composition/src/backend.rs) defines the interface for all backends:

#![allow(unused)]
fn main() {
pub trait CompositionBackend: Send + Sync {
    fn mount(&mut self) -> Result<()>;
    fn unmount(&mut self) -> Result<()>;
    fn status(&self) -> BackendStatus;
    fn cell_path(&self, cell: &str) -> Option<PathBuf>;
    fn refresh(&mut self) -> Result<()>;
}
}

Backend Selection

The selector (src/rust/composition/src/selector.rs) automatically chooses the appropriate backend:

#![allow(unused)]
fn main() {
pub fn select_backend(requested: BackendType) -> BackendSelection {
    match requested {
        BackendType::Auto => {
            if is_fuse_available() {
                BackendSelection::fuse("Auto-selected FUSE")
            } else {
                BackendSelection::symlink("Auto-selected symlinks (FUSE unavailable)")
            }
        }
        BackendType::Fuse => { /* ... */ }
        BackendType::Symlink => { /* ... */ }
    }
}
}

State Machine

The consistency state machine (src/rust/composition/src/state.rs) manages transitions:

Settled ──manifest change──► Syncing ──nix build──► Building
   ▲                                                    │
   │                                               build done
   │                                                    │
   └───────────────────── Transitioning ◄───────────────┘

Key types:

• ConsistencyStateMachine - Thread-safe state management
• StateObserver - Trait for state change notifications
• CellUpdate - Pending cell updates during transitions

Policy System

The policy system (src/rust/composition/src/policy.rs) controls access during updates. See FUSE Access Policy for details.

Layout System

Layouts (src/rust/composition/src/layout.rs) control how files are presented:

• Layout trait - Core interface for layouts
• LayoutRegistry - Runtime layout registration
• Buck2Layout - Default Buck2 layout
• BazelLayout - Bazel layout

See Custom Layouts for creating new layouts.

Module Structure

src/rust/nix-eval/src/          # Nix client abstraction (replaceable)
├── lib.rs                      # NixClient trait
├── cli.rs                      # CliNixClient (shells out to nix binary)
└── error.rs

src/rust/composition/src/
├── lib.rs                      # Public API exports
├── backend.rs                  # CompositionBackend trait
├── compose_config.rs           # compose.toml parser (single-mount legacy)
├── config.rs                   # CompositionConfig, CellConfig
├── discover.rs                 # Cell discovery via NixClient trait
├── error.rs                    # Error types
├── layout.rs                   # Layout system (Buck2Layout, BazelLayout)
├── policy.rs                   # Access policies
├── recovery.rs                 # Error recovery utilities
├── selector.rs                 # Backend selection logic
├── serve_config.rs             # Service config ([[mounts]] TOML format)
├── service.rs                  # Launchd/systemd service generation
├── state.rs                    # Consistency state machine
├── status.rs                   # BackendStatus enum
├── symlink.rs                  # Symlink backend
├── synthetic.rs                # macOS synthetic firmlink management
├── tracing.rs                  # Logging and debugging
├── watcher.rs                  # File watching (optional)
└── fuse/                       # FUSE backend (feature-gated)
    ├── mod.rs                  # Re-exports FuseBackend (platform-conditional)
    ├── fs_core.rs              # Platform-agnostic filesystem logic
    ├── platform.rs             # Platform detection and FUSE availability
    ├── filesystem.rs           # Linux: fuser Filesystem trait impl
    ├── backend.rs              # Linux: FuseBackend using fuser crate
    ├── edit_overlay.rs         # Copy-on-write editing layer
    ├── patch_generator.rs      # Unified diff generation for edits
    └── fuse_t/                 # macOS: direct libfuse-t backend
        ├── mod.rs
        ├── bindings.rs         # Hand-written FFI bindings to libfuse3
        ├── operations.rs       # FUSE operation callbacks (path-based API)
        └── backend.rs          # FuseTBackend implementing CompositionBackend

nix/home-manager/
└── turnkey-composed.nix        # Home-manager module for service management

Daemon Architecture

The turnkey-composed daemon supports two modes:

• start: Single mount, ad-hoc usage
• serve: Multi-mount service mode, reads config file, watches for changes

In service mode, the daemon:

1. Reads ~/.config/turnkey/composed.toml for mount declarations
2. For each mount: discovers cells via nix-eval crate, builds them, creates the FUSE mount
3. Watches manifest files for dependency changes (triggers cell rebuild)
4. Watches the config file for new/removed mounts (hot-reload)
5. On macOS, manages synthetic firmlinks for mount points under /

Nix Integration

Cell derivations are exposed as flake packages (godeps-cell, rustdeps-cell, etc.) by the flake-parts module. The daemon builds them via the NixClient trait (currently CliNixClient which shells out to nix). This abstraction allows replacing the CLI with a direct Nix daemon client when one becomes available.

FUSE Backend Implementation

The FUSE backend uses a layered architecture with a platform-agnostic core and platform-specific adapters.

FsCore (Platform-Agnostic)

FsCore (fs_core.rs) contains all filesystem logic with zero dependency on the fuser crate:

• Path resolution: resolve_path(path) -> ResolvedPath maps FUSE paths to logical locations (Root, Source, CellPrefix, Cell, VirtualFile, etc.)
• Inode management: Allocation, mapping, and lookup using plain u64 inode numbers
• Virtual file generation: .buckconfig and .buckroot content
• Policy checking: Access control during dependency updates
• Edit overlay: Copy-on-write editing of external dependencies

Both the Linux and macOS backends delegate to FsCore for all filesystem logic, converting between their own FUSE types and FsCore's neutral types.

Linux Backend (fuser crate)

Uses the fuser crate's low-level inode-based API:

• CompositionFs wraps FsCore and implements fuser::Filesystem
• Converts between fuser::INodeNo/FileAttr and FsCore's u64/FsAttr
• Feature flag: fuse (enables dep:fuser)

macOS Backend (FUSE-T FFI)

Uses direct C FFI to FUSE-T's libfuse3, bypassing the fuser crate entirely. This is necessary because fuser reads the FUSE file descriptor directly, which is incompatible with FUSE-T's NFS-based socket protocol.

• bindings.rs: Hand-written FFI bindings to libfuse3 (44-field fuse_operations struct at 352 bytes, fuse_new, fuse_mount, fuse_loop, etc.)
• operations.rs: extern "C" callbacks using the high-level path-based API. Each callback retrieves FsCore via a global AtomicPtr and delegates to resolve_path()
• backend.rs: FuseTBackend spawns a thread calling fuse_new + fuse_mount + fuse_loop
• Feature flag: fuse-t (only dep:libc needed)
• Links against /usr/local/lib/libfuse3.dylib (from FUSE-T)

FUSE-T quirks discovered during implementation:

• fuse_get_context()->private_data does not reliably pass the user_data from fuse_new. A global AtomicPtr<FsCore> is used instead.
• readdir filler must pass null for the stat buffer. FUSE-T's NFS translation rejects certain stat formats with "RPC struct is bad".
• The fuse_operations struct must include the newer statx and syncfs fields even if unused, to match the 352-byte C ABI.

Conditional Compilation

Platform selection happens at compile time:

#![allow(unused)]
fn main() {
// In fuse/mod.rs:
#[cfg(target_os = "linux")]
pub use backend::FuseBackend;           // fuser-based

#[cfg(target_os = "macos")]
pub use fuse_t::backend::FuseTBackend as FuseBackend;  // libfuse-t FFI
}

The selector.rs gates on #[cfg(any(feature = "fuse", feature = "fuse-t"))] so both feature flags enable the FUSE code path.

Platform Detection

Runtime FUSE availability checking in platform.rs:

• Linux: Checks for /dev/fuse
• macOS: Checks for FUSE-T bundle (/Library/Filesystems/fuse-t.fs) or library (/usr/local/lib/libfuse-t.dylib)

Recovery System

The recovery module (src/rust/composition/src/recovery.rs) provides:

Retry Logic

#![allow(unused)]
fn main() {
pub async fn retry_with_backoff<T, F, Fut>(
    config: &RetryConfig,
    operation: F,
) -> Result<T>
where
    F: Fn() -> Fut,
    Fut: Future<Output = Result<T>>,
}

Error Classification

#![allow(unused)]
fn main() {
pub fn is_transient_error(error: &Error) -> bool {
    matches!(error, Error::Timeout(_) | Error::PathUpdating(_) | ...)
}
}

Recovery Actions

#![allow(unused)]
fn main() {
pub enum RecoveryAction {
    Retry { delay: Duration },
    ForceUnmount,
    RestartDaemon,
    ManualIntervention { instructions: String },
}
}

Tracing and Debugging

The tracing module (src/rust/composition/src/tracing.rs) provides:

Configuration

#![allow(unused)]
fn main() {
pub struct TracingConfig {
    pub enable_fuse_ops: bool,
    pub enable_state_transitions: bool,
    pub enable_metrics: bool,
    pub log_level: String,
}
}

State Logger

Implements StateObserver to log state transitions:

#![allow(unused)]
fn main() {
impl StateObserver for StateLogger {
    fn on_state_change(&self, from: SystemState, to: SystemState) {
        info!("State: {:?} -> {:?}", from, to);
    }
}
}

Metrics

Tracks performance metrics:

• Operation counts (lookup, read, readdir, etc.)
• Latency histograms
• Cache hit rates

Debug Information

#![allow(unused)]
fn main() {
pub struct DebugInfo {
    pub backend_type: String,
    pub mount_point: Option<PathBuf>,
    pub cells: Vec<CellDebugInfo>,
    pub state: SystemState,
    pub metrics: Option<Metrics>,
}
}

Testing

Unit Tests

Each module has unit tests:

cargo test -p composition

Integration Tests

Test with actual FUSE mounts (requires FUSE):

# Linux
cargo test -p composition --features fuse -- --ignored

# macOS (FUSE-T)
cargo test -p composition --features fuse-t -- --ignored

Mock Backend

For testing without FUSE:

#![allow(unused)]
fn main() {
use composition::testing::MockBackend;

let backend = MockBackend::new()
    .with_cell("godeps", "/nix/store/xxx-godeps")
    .with_status(BackendStatus::Ready);
}

Feature Flags

The crate uses feature flags:

[features]
default = []
fuse = ["fuser"]       # Enable FUSE backend
watcher = ["notify"]   # Enable file watching

Error Handling

The Error enum in error.rs covers all failure modes:

#![allow(unused)]
fn main() {
pub enum Error {
    AlreadyMounted(PathBuf),
    NotMounted,
    MountPointInaccessible { path, source },
    CellNotFound(String),
    FuseUnavailable(String),
    // ...
}
}

Errors include recovery suggestions:

#![allow(unused)]
fn main() {
impl Error {
    pub fn is_transient(&self) -> bool { /* ... */ }
    pub fn recovery_suggestion(&self) -> Option<String> { /* ... */ }
}
}

Configuration

The CompositionConfig struct holds all settings:

#![allow(unused)]
fn main() {
pub struct CompositionConfig {
    pub mount_point: PathBuf,
    pub cells: HashMap<String, CellConfig>,
    pub consistency_mode: ConsistencyMode,
    pub layout: String,
}

pub struct CellConfig {
    pub source_path: PathBuf,
    pub editable: bool,
}
}

Related Documentation

• FUSE Access Policy - Access control during updates
• Custom Layouts - Creating build system layouts
• Architecture Proposal - Original design document

FUSE Access Policy System

The FUSE composition layer includes a pluggable access policy system that controls how file operations behave during dependency updates. This allows developers to tune the trade-off between consistency and availability based on their workflow.

Overview

When the composition system is updating (rebuilding Nix derivations), file access to dependency cells may need to be controlled. The policy system determines whether to:

• Allow the operation immediately
• Block until the system becomes stable
• Deny with an error (e.g., EAGAIN)
• Allow with stale data and log a warning

┌─────────────────────────────────────────────────────────────┐
│                     FUSE Operation                          │
│   (lookup, getattr, read, readdir, write, create, ...)      │
└─────────────────────────────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────┐
│                   Classify Request                          │
│  path/inode → FileClass                                     │
│  state machine → SystemState                                │
│  operation → OperationType                                  │
└─────────────────────────────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────┐
│                   Policy.check()                            │
│  (FileClass, SystemState, OperationType) → PolicyDecision   │
└─────────────────────────────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────┐
│                   Execute Decision                          │
│  Allow → proceed                                            │
│  Block → wait then retry                                    │
│  Deny → return errno                                        │
│  AllowStale → proceed with warning                          │
└─────────────────────────────────────────────────────────────┘

Core Concepts

File Classes

Files in the composition view are classified by their behavioral characteristics:

Key insight: SourcePassthrough and virtual files are always accessible regardless of system state. Only CellContent access is subject to policy decisions.

System States

The composition system transitions through these states:

Settled ──manifest change──► Syncing ──nix build──► Building
   ▲                                                    │
   │                                               build done
   │                                                    │
   └───────────────────── Transitioning ◄───────────────┘

Operation Types

Built-in Policies

StrictPolicy

Best for: CI pipelines, production builds where correctness is critical

Blocks all cell access during any update phase. Reads will never return stale data, but may block for the duration of the Nix build.

#![allow(unused)]
fn main() {
StrictPolicy::new()                    // Default 5-minute timeout
StrictPolicy::with_timeout(Duration::from_secs(120))  // Custom timeout
}

LenientPolicy

Best for: Interactive development where latency matters

Allows stale reads during syncing and building phases, only blocks during the brief transition phase.

#![allow(unused)]
fn main() {
LenientPolicy::new()
}

CIPolicy

Best for: CI/CD environments where blocking is undesirable

Never blocks - immediately returns EAGAIN if the operation would need to wait. The caller can retry or handle the error.

#![allow(unused)]
fn main() {
CIPolicy::new()
}

DevelopmentPolicy (Default)

Best for: Day-to-day development work

A balanced approach:

• Syncing: Allow stale reads (quick phase)
• Building: Block (wait for fresh data)
• Error: Allow stale (degrade gracefully)

#![allow(unused)]
fn main() {
DevelopmentPolicy::new()
}

Creating Custom Policies

Implement the AccessPolicy trait to create custom behavior:

#![allow(unused)]
fn main() {
use composition::policy::{
    AccessPolicy, FileClass, SystemState, OperationType, PolicyDecision,
};
use std::time::Duration;

pub struct MyPolicy {
    block_timeout: Duration,
}

impl AccessPolicy for MyPolicy {
    fn check(
        &self,
        class: &FileClass,
        state: SystemState,
        op: OperationType,
    ) -> PolicyDecision {
        // Source files always accessible
        if class.is_always_accessible() {
            return PolicyDecision::Allow;
        }

        // Custom logic based on state and operation
        match (state, op) {
            // Allow reads during syncing
            (SystemState::Syncing, OperationType::Read) => {
                PolicyDecision::AllowStale
            }
            // Block lookups during building
            (SystemState::Building, OperationType::Lookup) => {
                PolicyDecision::Block {
                    timeout: self.block_timeout,
                }
            }
            // Fail fast for directory listing during updates
            (_, OperationType::Readdir) if state.is_updating() => {
                PolicyDecision::eagain()
            }
            // Default: allow
            _ => PolicyDecision::Allow,
        }
    }

    fn name(&self) -> &'static str {
        "my-policy"
    }

    fn description(&self) -> &'static str {
        "Custom policy with special handling for readdir"
    }
}
}

Policy Decision Types

Convenience Constructors

#![allow(unused)]
fn main() {
PolicyDecision::block()                    // Block with 5-minute timeout
PolicyDecision::block_with_timeout(dur)    // Block with custom timeout
PolicyDecision::eagain()                   // Deny with EAGAIN (11)
PolicyDecision::ebusy()                    // Deny with EBUSY (16)
}

Configuring the Policy

In Rust Code

When creating a CompositionFs, use the with_policy constructor:

#![allow(unused)]
fn main() {
use composition::{CompositionConfig, CompositionFs};
use composition::policy::{CIPolicy, StrictPolicy};

// With CI policy
let fs = CompositionFs::with_policy(
    config,
    repo_root,
    state_machine,
    Box::new(CIPolicy::new()),
);

// With strict policy and custom timeout
let fs = CompositionFs::with_policy(
    config,
    repo_root,
    state_machine,
    Box::new(StrictPolicy::with_timeout(Duration::from_secs(60))),
);
}

Via Nix Configuration (Future)

turnkey.fuse = {
  enable = true;

  # Policy selection
  accessPolicy = "development";  # "strict" | "lenient" | "ci" | "development"

  # Custom timeout for blocking policies
  blockTimeout = 300;  # seconds
};

Environment Variables (Future)

# Override policy at runtime
TURNKEY_ACCESS_POLICY=ci tk build //...

# Custom timeout
TURNKEY_BLOCK_TIMEOUT=60 tk build //...

Debugging Policies

Policy decisions are logged at debug level. Enable debug logging to see:

DEBUG Policy 'development': blocking for up to 300s until stable
DEBUG Policy 'ci': denying Readdir on CellContent { cell: "godeps" } in state Building
WARN  Policy 'lenient': returning potentially stale data for cell 'godeps' during Building

Guidelines for Choosing a Policy

API Reference

Module: composition::policy

Types:

• FileClass - File classification enum
• SystemState - System state enum
• OperationType - Operation type enum
• PolicyDecision - Decision enum
• AccessPolicy - Policy trait
• BoxedPolicy - Type alias for Box<dyn AccessPolicy>

Built-in Policies:

• StrictPolicy
• LenientPolicy
• CIPolicy
• DevelopmentPolicy

Functions:

• default_policy() - Returns a boxed DevelopmentPolicy

Constants:

• EAGAIN - Resource temporarily unavailable (11)
• EBUSY - Device or resource busy (16)

Flake-Parts Module

Located at nix/flake-parts/turnkey/default.nix.

Purpose

Provides the user-facing API for Turnkey configuration in flakes.

Options Reference

turnkey.toolchains.enable

type = types.bool;
default = true;

Enable/disable Turnkey toolchain management.

turnkey.toolchains.declarationFiles

type = types.attrsOf types.path;
default = {};

Map shell names to toolchain.toml files:

declarationFiles = {
  default = ./toolchain.toml;
  ci = ./toolchain.ci.toml;
};

turnkey.toolchains.registry

type = types.lazyAttrsOf types.package;
default = {};

Custom toolchain registry. If empty, uses default registry.

turnkey.toolchains.wrapNativeTools

type = types.bool;
default = true;

Wrap go, cargo, uv with auto-sync behavior.

turnkey.toolchains.buck2

Nested options for Buck2 integration:

• enable - Enable Buck2 cell generation
• prelude.strategy - How to provide prelude ("nix", "bundled", "git", "path")
• go.enable, go.depsFile - Go dependency configuration
• rust.enable, rust.depsFile - Rust dependency configuration
• python.enable, python.depsFile - Python dependency configuration

Implementation

The module:

1. Imports default registry
2. Builds tw wrappers for native tools
3. Creates shell configurations for each declaration file
4. Passes configuration to devenv module

Extending

Add new options in the options.perSystem block and implement in config.perSystem.

Devenv Module

Located at nix/devenv/turnkey/default.nix.

Purpose

Configures individual devenv shells with toolchains and Buck2 integration.

Options

turnkey.enable

Enable Turnkey for this shell.

turnkey.declarationFile

Path to toolchain.toml file.

turnkey.registry

Package registry (usually inherited from flake-parts).

How It Works

1. Parse TOML: Reads toolchain.toml

toolchainDeclaration = builtins.fromTOML (builtins.readFile cfg.declarationFile);
toolchainNames = builtins.attrNames toolchainDeclaration.toolchains;

2. Resolve packages: Maps names to packages

resolvedPackages = map (name: cfg.registry.${name}) toolchainNames;

3. Add to shell: Packages added to devenv

config.packages = resolvedPackages;

Sub-Module: buck2.nix

The buck2.nix sub-module (nix/devenv/turnkey/buck2.nix) handles:

• Toolchains cell generation
• Prelude cell symlink
• Dependency cell symlinks
• Shell entry hooks

Shell Entry Hooks

Devenv's enterShell hook:

1. Symlinks .turnkey/prelude → Nix store
2. Symlinks .turnkey/toolchains → Nix store
3. Symlinks dependency cells if configured
4. Displays welcome message

Debugging

Enable verbose output:

TURNKEY_VERBOSE=1 nix develop

Buck2 Cell Generation

This document describes how Turnkey generates Buck2 cells from Nix derivations.

Overview

Turnkey generates several types of Buck2 cells:

• Toolchains cell - Language toolchains (Go, Rust, Python, etc.)
• Dependency cells - Third-party packages (godeps, rustdeps, pydeps)
• Prelude cell - Buck2 prelude with extensions

All cells are built as Nix derivations and symlinked into .turnkey/.

Toolchains Cell

Located at nix/buck2/toolchains-cell.nix. Generated from nix/buck2/mappings.nix.

Mapping Structure

{
  go = {
    skip = false;
    targets = [{
      name = "go";
      rule = "system_go_toolchain";
      load = "@prelude//toolchains/go:system_go_toolchain.bzl";
      visibility = [ "PUBLIC" ];
      dynamicAttrs = registry: {
        go_binary = "${registry.go}/bin/go";
      };
    }];
    implicitDependencies = [ "python" "cxx" ];
    runtimeDependencies = [ ];
  };
}

Generated Output

rules.star is generated with:

1. Load statements for each rule
2. Rule instantiations with configured attributes
3. Visibility set to PUBLIC

Adding Toolchain Mappings

Edit nix/buck2/mappings.nix:

mylang = {
  skip = false;
  targets = [{
    name = "mylang";
    rule = "system_mylang_toolchain";
    load = "@prelude//mylang:toolchain.bzl";
    visibility = [ "PUBLIC" ];
    dynamicAttrs = registry: {
      compiler_path = "${registry.mylang}/bin/mylang";
    };
  }];
  implicitDependencies = [ ];
  runtimeDependencies = [ ];
};

Go Dependency Cell

Built by nix/buck2/go-deps-cell.nix.

Cell Structure

/nix/store/<hash>-go-deps-cell/
├── .buckconfig           # Cell identity
├── rules.star            # Root rules.star file with package list
└── vendor/
    └── github.com/
        └── spf13/
            └── cobra/
                ├── rules.star    # go_library target
                └── *.go          # Source files from Nix

The directory structure mirrors Go import paths, matching Buck2's conventions for third-party Go packages.

Cell Configuration

# .buckconfig
[cells]
    godeps = .
    prelude = bundled://

[buildfile]
    name = rules.star

Generated rules.star Files

Each package gets a go_library target:

# vendor/github.com/spf13/cobra/rules.star
go_library(
    name = "cobra",
    srcs = glob(["*.go"], exclude = ["*_test.go"]),
    importpath = "github.com/spf13/cobra",
    deps = [
        "//vendor/github.com/spf13/pflag:pflag",
        "//vendor/github.com/inconshreveable/mousetrap:mousetrap",
    ],
    visibility = ["PUBLIC"],
)

Target Path Format

When writing rules.star files that depend on packages from the godeps cell:

godeps//vendor/<import-path>:<target-name>

Where:

• godeps// - the cell alias (configured in .buckconfig)
• vendor/ - required prefix - all packages live under vendor/
• <import-path> - the full Go import path
• <target-name> - the directory name (last path component), NOT the package name

Examples:

Common Mistakes:

# WRONG - missing vendor/ prefix
deps = ["godeps//github.com/spf13/cobra:cobra"]

# WRONG - using package name instead of directory name for versioned imports
deps = ["godeps//vendor/github.com/pelletier/go-toml/v2:go-toml"]

# CORRECT
deps = ["godeps//vendor/github.com/pelletier/go-toml/v2:v2"]

Import Path Resolution

Buck2's importpath attribute ensures the Go compiler sees the correct import path:

• Buck2 target: godeps//vendor/github.com/spf13/cobra:cobra
• Go import: import "github.com/spf13/cobra"

The go_library rule's importpath = "github.com/spf13/cobra" makes this work.

Nix Integration

# nix/buck2/go-deps-cell.nix
{ pkgs, lib, goDepsFile }:

let
  # Parse go-deps.toml to get dependencies
  deps = builtins.fromTOML (builtins.readFile goDepsFile);

  # Fetch each dependency source
  depSources = lib.mapAttrs (name: info:
    pkgs.fetchFromGitHub {
      owner = info.owner;
      repo = info.repo;
      rev = info.rev;
      hash = info.hash;
    }
  ) deps.deps;

  # Generate rules.star file content for each dep
  generateBuck = name: src: ''
    go_library(
      name = "${lib.last (lib.splitString "/" name)}",
      srcs = glob(["*.go"], exclude = ["*_test.go"]),
      importpath = "${name}",
      deps = [${formatDeps (getDeps name)}],
      visibility = ["PUBLIC"],
    )
  '';
in
pkgs.runCommand "go-deps-cell" {} ''
  mkdir -p $out/vendor

  ${lib.concatStrings (lib.mapAttrsToList (name: src: ''
    mkdir -p $out/vendor/${name}
    cp -r ${src}/* $out/vendor/${name}/
    cat > $out/vendor/${name}/rules.star <<'EOF'
    ${generateBuck name src}
    EOF
  '') depSources)}

  # Generate cell .buckconfig
  cat > $out/.buckconfig <<'EOF'
  [cells]
      godeps = .
  EOF
''

Rust Dependency Cell

Built by nix/buck2/rust-deps-cell.nix.

Process

1. Reads rust-deps.toml
2. Fetches crates from crates.io
3. Computes unified features across dependency graph
4. Generates rules.star with features and deps

Special Handling

Rust crates may require:

• rustc flags - Build scripts that emit cargo:rustc-cfg
• Generated files - Build scripts that generate .rs files
• Native code - Build scripts that compile C/assembly

See Dependency Generators for handling these cases.

Python Dependency Cell

Built by nix/buck2/python-deps-cell.nix.

Process

1. Reads python-deps.toml
2. Fetches wheels from PyPI
3. Generates rules.star per package

Cell Configuration

Each cell gets a .buckconfig:

[cells]
    cellname = .
    prelude = path/to/prelude

[buildfile]
    name = rules.star

Dual-Build Compatibility

A key design goal is that code builds with both native tools and Buck2:

# Native build (uses go.mod directly)
go build ./...

# Buck2 build (uses generated cell)
buck2 build //...

This is achieved by:

1. No import rewriting - Go code uses standard import paths (github.com/foo/bar)
2. importpath attribute - Buck2's go_library rule's importpath tells the compiler the correct path
3. Nix-managed deps - Dependencies fetched by Nix, not vendored in repo

Adding New Dependency Cell Types

To add support for a new language:

1. Create deps generator (e.g., newlang-deps-gen)
2. Create cell builder (nix/buck2/newlang-deps-cell.nix)
3. Add to devenv module (nix/devenv/turnkey/buck2.nix)
4. Add configuration options for deps file path

Debugging

Inspect Generated rules.star Files

cat .turnkey/godeps/vendor/github.com/spf13/cobra/rules.star

Check Cell Contents

ls -la .turnkey/godeps/vendor/

Verify Cell Configuration

cat .turnkey/godeps/.buckconfig

List Available Targets

buck2 targets godeps//...

Adding Toolchains

This guide covers adding new toolchains to Turnkey.

Steps

1. Add package to registry (versioned format)
2. Add mapping to mappings.nix
3. (Optional) Create prelude extension

1. Add to Registry

For standard toolchains (available in nixpkgs), contribute to teller's registry/default.nix.

For project-specific tools, use registryExtensions in your flake.nix:

turnkey.toolchains = {
  registryExtensions = let
    single = pkg: { versions = { "default" = pkg; }; default = "default"; };
  in {
    # Single version (most common)
    zig = single pkgs.zig;

    # Multiple versions
    nodejs = {
      versions = {
        "18" = pkgs.nodejs_18;
        "20" = pkgs.nodejs_20;
        "22" = pkgs.nodejs_22;
      };
      default = "20";
    };
  };
};

Versioned Format

Each registry entry must have:

<name> = {
  versions = { "<version>" = <derivation>; ... };
  default = "<version>";  # Must match a key in versions
};

The single helper is convenient for tools with only one version.

2. Add Toolchain Mapping

Edit nix/buck2/mappings.nix:

For Standard Toolchains

zig = {
  skip = false;
  targets = [{
    name = "zig";
    rule = "system_zig_toolchain";
    load = "@prelude//zig:toolchain.bzl";
    visibility = [ "PUBLIC" ];
  }];
  implicitDependencies = [ ];
};

For Non-Toolchain Tools

Some tools don't need Buck2 rules:

mydevtool = {
  skip = true;
  reason = "Development utility, not a Buck2 toolchain";
};

3. Dynamic Attributes

For toolchains needing Nix store paths, use dynamicAttrs. The function receives a resolved registry where entries are already derivations:

mylang = {
  targets = [{
    name = "mylang";
    rule = "system_mylang_toolchain";
    load = "@prelude//mylang:toolchain.bzl";
    # registry entries are already resolved to derivations
    dynamicAttrs = registry: {
      compiler = "${registry.mylang}/bin/mycompiler";
    };
  }];
};

4. Prelude Extension (if needed)

If the upstream prelude doesn't have rules for your language, create a prelude extension. See Prelude Extensions.

Testing

1. Add toolchain to toolchain.toml
2. Stage files: git add nix/
3. Enter shell: nix develop
4. Verify: tk targets toolchains//...

Creating Custom Registries

For reusable toolchain collections, create a registry flake:

# my-registry/flake.nix
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    turnkey.url = "github:firefly-engineering/turnkey";
  };

  outputs = { nixpkgs, turnkey, ... }:
    let
      forAllSystems = f: nixpkgs.lib.genAttrs
        [ "x86_64-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin" ]
        (system: f system);
    in {
      overlays.default = forAllSystems (system:
        turnkey.lib.${system}.mkRegistryOverlay (final: prev: {
          # Add toolchains
          zig = {
            versions = {
              "0.11" = final.zig_0_11;
              "0.12" = final.zig_0_12;
            };
            default = "0.12";
          };

          # Extend existing toolchain with more versions
          go = {
            versions = { "1.24" = final.go_1_24; };
            default = "1.24";
          };
        })
      );
    };
}

Consumers compose the overlay:

pkgs = import nixpkgs {
  overlays = [
    my-registry.overlays.default.${system}
  ];
};

When multiple registries are composed:

• Versions merge additively - all versions from all registries are available
• Default is overridden - later overlays set the default version

Prelude Extensions

This document covers how to add custom Buck2 rules to the prelude and the various customization approaches available.

Overview

The Buck2 prelude is a collection of Starlark rules that provide build functionality for various languages (Go, Rust, Python, C++, etc.). It's the "standard library" of build rules that ships with Buck2.

Prelude extensions live in nix/buck2/prelude-extensions/. They're copied into the prelude during the Nix build.

Directory Structure

nix/buck2/prelude-extensions/
└── mylang/
    ├── providers.bzl     # Provider definitions
    ├── toolchain.bzl     # Toolchain rule
    ├── mylang_library.bzl
    ├── mylang_binary.bzl
    └── mylang.bzl        # Convenience exports

Creating an Extension

1. Create Provider

providers.bzl:

MylangToolchainInfo = provider(
    doc = "Mylang toolchain information.",
    fields = {
        "compiler": provider_field(typing.Any, default = None),
    },
)

MylangLibraryInfo = provider(
    doc = "Information about a mylang library.",
    fields = {
        "output": provider_field(typing.Any, default = None),
    },
)

2. Create Toolchain Rule

toolchain.bzl:

load(":providers.bzl", "MylangToolchainInfo")

def _system_mylang_toolchain_impl(ctx):
    compiler_path = ctx.attrs.compiler_path
    return [
        DefaultInfo(),
        MylangToolchainInfo(
            compiler = RunInfo(args = cmd_args(compiler_path)),
        ),
    ]

system_mylang_toolchain = rule(
    impl = _system_mylang_toolchain_impl,
    attrs = {
        "compiler_path": attrs.string(
            doc = "Path to the mylang compiler binary",
        ),
    },
    is_toolchain_rule = True,
    doc = "System-provided mylang toolchain.",
)

3. Create Build Rules

mylang_binary.bzl:

load(":providers.bzl", "MylangToolchainInfo")

def _mylang_binary_impl(ctx):
    toolchain = ctx.attrs._toolchain[MylangToolchainInfo]
    out = ctx.actions.declare_output(ctx.label.name)

    ctx.actions.run(
        cmd_args(
            toolchain.compiler.args,
            ctx.attrs.srcs,
            "-o",
            out.as_output(),
        ),
        category = "mylang_compile",
        identifier = ctx.label.name,
    )

    return [
        DefaultInfo(default_output = out),
        RunInfo(args = cmd_args(out)),
    ]

mylang_binary = rule(
    impl = _mylang_binary_impl,
    attrs = {
        "srcs": attrs.list(
            attrs.source(),
            doc = "Source files to compile",
        ),
        "_toolchain": attrs.toolchain_dep(
            default = "toolchains//:mylang",
            providers = [MylangToolchainInfo],
        ),
    },
    doc = "Build a mylang executable.",
)

4. Export Rules

mylang.bzl:

load(":mylang_binary.bzl", _mylang_binary = "mylang_binary")
load(":mylang_library.bzl", _mylang_library = "mylang_library")
load(":toolchain.bzl", _system_mylang_toolchain = "system_mylang_toolchain")
load(":providers.bzl", _MylangToolchainInfo = "MylangToolchainInfo")

mylang_binary = _mylang_binary
mylang_library = _mylang_library
system_mylang_toolchain = _system_mylang_toolchain
MylangToolchainInfo = _MylangToolchainInfo

5. Add Toolchain Mapping

Edit nix/buck2/mappings.nix:

mylang = {
  skip = false;
  targets = [{
    name = "mylang";
    rule = "system_mylang_toolchain";
    load = "@prelude//mylang:toolchain.bzl";
    visibility = [ "PUBLIC" ];
    dynamicAttrs = registry: {
      compiler_path = "${registry.mylang}/bin/mylang";
    };
  }];
  implicitDependencies = [ ];
  runtimeDependencies = [ ];
};

Building

Extensions are included when you rebuild the prelude:

git add nix/buck2/prelude-extensions/mylang/
nix build .#turnkey-prelude

Customization Approaches

Approach 1: Extension Cell Pattern

Create a separate cell for custom rules alongside the standard prelude:

project/
├── prelude/           # Standard prelude (submodule or external)
├── prelude-custom/    # Custom extensions
│   ├── BUCK
│   ├── platforms/
│   ├── toolchains/
│   └── rules/
└── .buckconfig

[cells]
prelude = prelude
prelude-custom = prelude-custom

[external_cells]
prelude = bundled

[build]
execution_platforms = prelude-custom//platforms:default

Pros:

• Clean separation of concerns
• Can still use bundled prelude for core rules
• Easy to track what's custom vs standard
• No fork maintenance burden

Cons:

• Two cells to manage
• Must understand which rules come from where

Approach 2: Custom Rules Outside Prelude

Define rules anywhere in your project - they don't need to be in the prelude:

# rules/my_rules.bzl
def my_custom_rule_impl(ctx):
    # Implementation
    pass

my_custom_rule = rule(
    impl = my_custom_rule_impl,
    attrs = {
        "src": attrs.source(),
        "deps": attrs.list(attrs.dep()),
    },
)

# BUCK
load("//rules:my_rules.bzl", "my_custom_rule")

my_custom_rule(
    name = "my_target",
    src = "input.txt",
)

Pros:

• No prelude modification needed
• Explicit load() makes dependencies clear
• Rules live with the project

Cons:

• Must use explicit load() statements
• Not globally available like prelude rules

Approach 3: Nix-Backed Prelude Cell (Recommended)

This is Turnkey's recommended approach. The prelude Nix derivation:

1. Fetches upstream prelude from buck2-prelude repository
2. Applies turnkey patches for customizations
3. Adds custom rules from nix/buck2/prelude-extensions/

# nix/buck2/prelude.nix
{ pkgs, lib }:

let
  upstreamPrelude = pkgs.fetchFromGitHub {
    owner = "facebook";
    repo = "buck2-prelude";
    rev = "...";  # Pinned commit
    hash = "sha256-...";
  };
in
pkgs.runCommand "turnkey-prelude" {} ''
  cp -r ${upstreamPrelude} $out
  chmod -R u+w $out

  # Apply turnkey patches
  patch -d $out -p1 < ${../patches/prelude/nix-integration.patch}

  # Add custom rules
  cp -r ${./prelude-extensions}/* $out/
''

Advantages:

Approach 4: Forked Prelude

Maintain a fork of the Buck2 prelude with your modifications.

[external_cells]
prelude = git

[external_cell_prelude]
git_origin = https://github.com/your-org/buck2-prelude-fork.git
commit_hash = your-fork-commit-hash

Pros:

• Complete control over all rules
• Can modify any prelude behavior

Cons:

• Significant maintenance burden
• Must track upstream changes
• Risk of divergence from upstream

Prelude Version Compatibility

The Buck2 binary and its prelude must be version-matched. Using a mismatched prelude can cause cryptic Starlark errors.

Symptoms

Finding Compatible Versions

1. Check buck2 version:

   buck2 --version
   # Output: buck2 2025-12-01-75e4243c93877a3db4acf55f20d2e80a32523233
   

2. Find matching prelude commit (same date or slightly before):

   curl -s "https://api.github.com/repos/facebook/buck2-prelude/commits?until=2025-12-02T00:00:00Z&per_page=5" | \
     jq -r '.[] | "\(.sha) \(.commit.committer.date)"'
   

3. Update nix/buck2/prelude.nix with new rev and hash.

When to Update

Update both buck2 and prelude when:

• nixpkgs updates buck2
• You need new prelude features
• Build errors appear after nixpkgs update

Existing Extensions

Turnkey includes these prelude extensions:

• typescript/ - TypeScript compiler integration
• mdbook/ - Documentation builder

When to Customize

Consider prelude customization when:

1. Built-in rules don't support your workflow - e.g., Nix-specific build patterns
2. You need enhanced toolchain control - beyond what system toolchains provide
3. Platform definitions need modification - custom constraint values
4. You're integrating with external systems - CI/CD, remote execution

References

• Buck2 External Cells Documentation
• Buck2 Writing Rules
• Buck2 Writing Toolchains
• Buck2 Prelude Repository

Custom Rules

Create custom Buck2 rules for your project.

Project-Local Rules

For rules specific to your project, create a rules/ directory:

my-project/
├── rules/
│   ├── myrule.bzl
│   └── rules.star
└── rules.star

In rules/rules.star:

# Export rules from this cell
load(":myrule.bzl", "my_rule")

Reference from your project:

load("//rules:myrule.bzl", "my_rule")

my_rule(
    name = "example",
    # ...
)

Prelude-Level Rules

For rules you want available across multiple projects, add them as prelude extensions. See Prelude Extensions.

Rule Structure

Basic Rule

def _my_rule_impl(ctx: AnalysisContext) -> list[Provider]:
    out = ctx.actions.declare_output("output.txt")

    ctx.actions.run(
        cmd_args("echo", "hello", ">", out.as_output()),
        category = "my_rule",
    )

    return [DefaultInfo(default_output = out)]

my_rule = rule(
    impl = _my_rule_impl,
    attrs = {
        "srcs": attrs.list(attrs.source()),
    },
)

With Toolchain

def _my_rule_impl(ctx):
    toolchain = ctx.attrs._toolchain[MyToolchainInfo]
    # Use toolchain...

my_rule = rule(
    impl = _my_rule_impl,
    attrs = {
        "_toolchain": attrs.toolchain_dep(
            default = "toolchains//:mytool",
            providers = [MyToolchainInfo],
        ),
    },
)

With RunInfo

def _my_binary_impl(ctx):
    out = ctx.actions.declare_output(ctx.label.name)
    # Build the binary...

    run_info = RunInfo(args = cmd_args(out))

    return [
        DefaultInfo(default_output = out),
        run_info,  # Makes it runnable with `buck2 run`
    ]

Best Practices

1. Use categories in ctx.actions.run() for build output
2. Declare all outputs explicitly
3. Use hidden deps for non-output dependencies
4. Provide sensible defaults for optional attrs

Custom Layouts

The composition layer uses a pluggable layout system to support different build systems. While Turnkey ships with Buck2 and Bazel layouts, you can create custom layouts for other build systems or specialized requirements.

Layout Architecture

Layouts control how the composed filesystem presents dependencies:

┌─────────────────────────────────────────────────────────────────┐
│                   Layout System                                 │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  LayoutContext ──────────────► Layout.map_dep()                 │
│  (mount point,                   │                              │
│   repo root,                     ▼                              │
│   cells)            /firefly/project/external/godeps/vendor/... │
│                                                                 │
│  LayoutContext ──────────────► Layout.generate_config()         │
│                                  │                              │
│                                  ▼                              │
│                           .buckconfig, .buckroot, etc.          │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

The Layout Trait

All layouts implement the Layout trait:

#![allow(unused)]
fn main() {
use composition::layout::{Layout, LayoutContext, ConfigFile, CellInfo};
use std::path::{Path, PathBuf};

pub trait Layout: Send + Sync {
    /// Layout name (e.g., "buck2", "bazel", "custom")
    fn name(&self) -> &'static str;

    /// Map a dependency path to its composed location
    fn map_dep(&self, ctx: &LayoutContext, cell: &str, path: &Path) -> Option<PathBuf>;

    /// Generate configuration files for this build system
    fn generate_config(&self, ctx: &LayoutContext) -> Vec<ConfigFile>;

    /// List of cells this layout supports
    fn supported_cells(&self, ctx: &LayoutContext) -> Vec<String>;
}
}

LayoutContext

The LayoutContext provides all information needed for layout operations:

#![allow(unused)]
fn main() {
pub struct LayoutContext {
    /// Mount point (e.g., "/firefly/turnkey")
    pub mount_point: PathBuf,

    /// Repository root (actual filesystem path)
    pub repo_root: PathBuf,

    /// Name of the source overlay directory (default: "root")
    pub source_dir_name: String,

    /// Prefix for cell directories (default: "external")
    pub cell_prefix: String,

    /// Available cells
    pub cells: Vec<CellInfo>,
}

pub struct CellInfo {
    /// Cell name (e.g., "godeps")
    pub name: String,

    /// Source path (Nix store path)
    pub source_path: PathBuf,

    /// Whether editing is enabled
    pub editable: bool,
}
}

Helper Methods

#![allow(unused)]
fn main() {
impl LayoutContext {
    /// Get the path to a cell's directory
    /// e.g., "/firefly/turnkey/external/godeps"
    pub fn cell_path(&self, cell: &str) -> PathBuf;

    /// Get the root source directory path
    /// e.g., "/firefly/turnkey/root"
    pub fn source_path(&self) -> PathBuf;

    /// Check if a cell exists
    pub fn has_cell(&self, name: &str) -> bool;
}
}

Creating a Custom Layout

Basic Example

#![allow(unused)]
fn main() {
use composition::layout::{Layout, LayoutContext, ConfigFile};
use std::path::{Path, PathBuf};

pub struct PleaseLayout;

impl Layout for PleaseLayout {
    fn name(&self) -> &'static str {
        "please"
    }

    fn map_dep(&self, ctx: &LayoutContext, cell: &str, path: &Path) -> Option<PathBuf> {
        // Map cell paths to Please's third_party structure
        // e.g., godeps -> third_party/go
        let target_dir = match cell {
            "godeps" => "third_party/go",
            "rustdeps" => "third_party/rust",
            "pydeps" => "third_party/python",
            _ => return None,
        };
        Some(ctx.mount_point.join(target_dir).join(path))
    }

    fn generate_config(&self, ctx: &LayoutContext) -> Vec<ConfigFile> {
        // Generate .plzconfig at the root
        let config = format!(
            r#"[please]
version = >=17.0.0

[build]
path = {}

[go]
importpath = github.com/example/project
"#,
            ctx.source_path().display()
        );

        vec![ConfigFile::new(".plzconfig", config)]
    }

    fn supported_cells(&self, ctx: &LayoutContext) -> Vec<String> {
        ctx.cells
            .iter()
            .filter(|c| matches!(c.name.as_str(), "godeps" | "rustdeps" | "pydeps"))
            .map(|c| c.name.clone())
            .collect()
    }
}
}

Using SimpleLayout

For quick prototyping, use SimpleLayout without implementing the full trait:

#![allow(unused)]
fn main() {
use composition::layout::{SimpleLayout, LayoutContext, ConfigFile};

let layout = SimpleLayout::new(
    "pants",
    |ctx, cell, path| {
        // Custom path mapping
        Some(ctx.mount_point.join("3rdparty").join(cell).join(path))
    },
    |ctx| {
        // Generate config files
        vec![
            ConfigFile::new("pants.toml", "[GLOBAL]\npants_version = \"2.18.0\""),
            ConfigFile::new("BUILD", "# Root BUILD file"),
        ]
    },
);
}

Registering Custom Layouts

Using the Global Registry

Register your layout at application startup:

#![allow(unused)]
fn main() {
use composition::layout::{global_registry, BoxedLayout};

fn register_layouts() {
    let registry = global_registry();
    registry.register(Box::new(PleaseLayout));
    registry.register(Box::new(PantsLayout::new()));
}
}

Using a Custom Registry

For more control, create your own registry:

#![allow(unused)]
fn main() {
use composition::layout::{LayoutRegistry, BoxedLayout};

let mut registry = LayoutRegistry::new();
registry.register(Box::new(PleaseLayout));
registry.register(Box::new(CustomLayout::with_options(opts)));

// Look up by name
let layout = registry.get("please").expect("layout not found");

// List available layouts
for name in registry.available() {
    println!("Layout: {}", name);
}
}

ConfigFile

Generated configuration files use the ConfigFile struct:

#![allow(unused)]
fn main() {
pub struct ConfigFile {
    /// Relative path within the composed view
    pub path: PathBuf,

    /// File content
    pub content: String,
}

impl ConfigFile {
    pub fn new(path: impl Into<PathBuf>, content: impl Into<String>) -> Self;
}
}

Common patterns:

#![allow(unused)]
fn main() {
// Root config file
ConfigFile::new(".buckconfig", "...")

// Nested path
ConfigFile::new("build/config.bzl", "...")

// Per-cell config
ConfigFile::new(format!("{}/{}/BUILD", ctx.cell_prefix, cell), "...")
}

Layout Selection

Layouts are selected via configuration:

Nix Configuration

turnkey.fuse = {
  enable = true;
  layout = "please";  # Use custom layout
};

Runtime Selection

#![allow(unused)]
fn main() {
use composition::layout::{layout_by_name, default_layout};

// Get specific layout
let layout = layout_by_name("please")?;

// Or use default (buck2)
let layout = default_layout();
}

Available Layouts

#![allow(unused)]
fn main() {
use composition::layout::available_layouts;

for name in available_layouts() {
    println!("Available: {}", name);
}
}

Built-in Layouts

Buck2Layout

The default layout for Buck2 projects:

• Maps cells to external/<cell>/
• Generates .buckconfig with cell mappings
• Generates .buckroot marker

#![allow(unused)]
fn main() {
use composition::layout::Buck2Layout;

let layout = Buck2Layout::new();
// or with custom prelude cell
let layout = Buck2Layout::with_prelude("custom-prelude");
}

BazelLayout

For Bazel-based projects:

• Maps cells to external/<cell>/
• Generates WORKSPACE file
• Generates root BUILD.bazel

#![allow(unused)]
fn main() {
use composition::layout::BazelLayout;

let layout = BazelLayout::new();
}

Testing Custom Layouts

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;
    use composition::layout::{LayoutContext, CellInfo};
    use std::path::PathBuf;

    fn test_context() -> LayoutContext {
        LayoutContext {
            mount_point: PathBuf::from("/firefly/test"),
            repo_root: PathBuf::from("/home/user/project"),
            source_dir_name: "root".to_string(),
            cell_prefix: "external".to_string(),
            cells: vec![
                CellInfo {
                    name: "godeps".to_string(),
                    source_path: PathBuf::from("/nix/store/xxx-godeps"),
                    editable: false,
                },
            ],
        }
    }

    #[test]
    fn test_map_dep() {
        let layout = PleaseLayout;
        let ctx = test_context();

        let mapped = layout.map_dep(&ctx, "godeps", Path::new("vendor/foo"));
        assert_eq!(
            mapped,
            Some(PathBuf::from("/firefly/test/third_party/go/vendor/foo"))
        );
    }

    #[test]
    fn test_generate_config() {
        let layout = PleaseLayout;
        let ctx = test_context();

        let configs = layout.generate_config(&ctx);
        assert_eq!(configs.len(), 1);
        assert_eq!(configs[0].path, PathBuf::from(".plzconfig"));
        assert!(configs[0].content.contains("[please]"));
    }

    #[test]
    fn test_supported_cells() {
        let layout = PleaseLayout;
        let ctx = test_context();

        let cells = layout.supported_cells(&ctx);
        assert!(cells.contains(&"godeps".to_string()));
    }
}
}

Best Practices

1. Keep map_dep simple - Just path manipulation, no I/O
2. Generate minimal configs - Only what the build system needs
3. Support all standard cells - godeps, rustdeps, pydeps, jsdeps
4. Use cell_path() helper - For consistent path construction
5. Test with real build systems - Verify generated configs work
6. Document cell expectations - What each cell should contain

API Reference

Module: composition::layout

Traits:

• Layout - Core layout trait

Structs:

• LayoutContext - Context for layout operations
• LayoutRegistry - Registry for custom layouts
• CellInfo - Information about a cell
• ConfigFile - Generated configuration file
• SimpleLayout - Quick layout without full trait impl
• Buck2Layout - Built-in Buck2 layout
• BazelLayout - Built-in Bazel layout

Functions:

• global_registry() - Get the global layout registry
• available_layouts() - List available layout names
• layout_by_name(name) - Get a layout by name
• default_layout() - Get the default layout (Buck2)

Type Aliases:

• BoxedLayout - Box<dyn Layout>
• LayoutFactory - fn() -> BoxedLayout

Dependency Generators

Tools that generate deps TOML files from native lock files.

Overview

Each language has a generator that:

1. Reads native lock files (go.sum, Cargo.lock, uv.lock)
2. Extracts dependency information
3. Prefetches packages to get Nix hashes
4. Outputs deps TOML for Nix cell building

Generator Structure

Input

Native lock file format (varies by language).

Output

TOML file with dependencies:

# go-deps.toml example
[deps]
[deps."github.com/pkg/errors"]
version = "v0.9.1"
hash = "sha256-xyz..."

[deps."golang.org/x/sys"]
version = "v0.15.0"
hash = "sha256-abc..."

Existing Generators

godeps-gen (Go)

Located at cmd/godeps-gen/.

godeps-gen --prefetch -o go-deps.toml

Reads: go.mod, go.sum

rustdeps-gen (Rust)

Located at cmd/rustdeps-gen/.

rustdeps-gen --cargo-lock Cargo.lock -o rust-deps.toml

Reads: Cargo.lock

pydeps-gen (Python)

Located at cmd/pydeps-gen/.

pydeps-gen --lock pylock.toml -o python-deps.toml

Reads: pylock.toml, uv.lock, or requirements.txt

Rust Dependency Handling

Rust crates can have build.rs scripts that run during compilation. Since we can't run arbitrary code in Nix's sandbox, build script outputs must be handled manually.

The Standard Flow

Cargo.lock → rust-deps.toml → rust-deps-cell.nix → .turnkey/rustdeps/

1. Cargo.lock defines exact versions and dependency graph
2. rust-deps.toml adds Nix hashes for each crate (generated by rustdeps-gen)
3. rust-deps-cell.nix fetches crates and generates rules.star files
4. gen-rust-buck.py parses each crate's Cargo.toml to generate its rules.star

What Works Automatically

• Dependencies: Resolved from [dependencies] in Cargo.toml
• Features: Unified across the dependency graph (like Cargo does)
• Crate renaming: package = "real-name" in dependencies
• Proc-macros: Detected from [lib] proc-macro = true
• Edition: Read from package.edition (defaults to 2015)
• Crate root: Detected from [lib] path or standard locations

What Requires Manual Handling

Diagnosing Problems

Symptom: Undefined cfg Flag

Error:

error[E0425]: cannot find value `fast_arithmetic` in this scope

Diagnosis: The crate's build.rs sets this via cargo:rustc-cfg=fast_arithmetic="64".

Solution:

rustcFlagsRegistry = {
  serde_json = ["--cfg" ''fast_arithmetic=\"64\"''];
};

Symptom: Missing Generated File

Error:

error[E0432]: unresolved import `crate::private`

Diagnosis: The crate expects a file in OUT_DIR that build.rs generates.

Solution:

buildScriptFixups = {
  serde = { patchVersion, vendorPath, ... }: ''
    mkdir -p "$out/${vendorPath}/out_dir"
    cat > "$out/${vendorPath}/out_dir/private.rs" << 'EOF'
#[doc(hidden)]
pub mod __private${patchVersion} {
    pub use crate::private::*;
}
EOF
  '';
};

Symptom: Linker Error for Native Symbols

Error:

error: linking with `cc` failed: exit status: 1
  = note: undefined reference to `ring_core_0_17_14__OPENSSL_cpuid_setup'

Diagnosis: The crate has C/assembly code that build.rs compiles.

Solution: Complex fixup that compiles the native code (see ring example in defaults).

The Registry System

Configuration lives in your flake.nix under turnkey.toolchains.buck2:

{
  turnkey.toolchains = {
    buck2 = {
      enable = true;
      rustDepsFile = ./rust-deps.toml;

      # Rustc flags for build scripts that emit cfg directives
      rustcFlagsRegistry = {
        my_crate = ["--cfg" "my_flag"];
        "my_crate@1.2.3" = ["--cfg" "version_specific_flag"];
      };

      # Build script fixups for generated files
      buildScriptFixups = {
        my_crate = { patchVersion, vendorPath, ... }: ''
          mkdir -p "$out/${vendorPath}/out_dir"
          echo "// generated" > "$out/${vendorPath}/out_dir/generated.rs"
        '';
      };

      # Feature overrides (in separate file)
      rustFeaturesFile = ./rust-features.toml;
    };
  };
}

Version-Aware Lookup

Both registries support version-specific keys:

rustcFlagsRegistry = {
  # Catch-all for any version
  rustix = ["--cfg" "libc" "--cfg" "linux_like" "--cfg" "linux_kernel"];

  # Specific version override (takes precedence)
  "rustix@0.38.0" = ["--cfg" "libc" "--cfg" "linux_like"];
};

Resolution order:

1. "crate@version" - Exact versioned key
2. "crate" - Catch-all for any version
3. Default registry (if exists)
4. Empty (no flags/fixup)

Fixup Function Context

Fixup functions receive context about the crate:

buildScriptFixups = {
  my_crate = { crateName, version, patchVersion, key, vendorPath }: ''
    # crateName: "my_crate"
    # version: "1.2.3"
    # patchVersion: "3" (last component)
    # key: "my_crate@1.2.3"
    # vendorPath: "vendor/my_crate@1.2.3"

    echo "Building fixup for ${crateName} version ${version}"
    mkdir -p "$out/${vendorPath}/out_dir"
  '';
};

Nix Interpolation vs Shell Escaping

In Nix multiline strings ('' ... ''):

buildScriptFixups = {
  my_crate = { patchVersion, vendorPath, ... }: ''
    # CORRECT: ${patchVersion} is Nix interpolation
    MY_VAR="${patchVersion}"

    # WRONG: ''${patchVersion} escapes the $ for shell
    # This becomes literal ${patchVersion}, which is undefined
    MY_VAR="''${patchVersion}"  # Results in empty string!

    # CORRECT: $out is a shell variable (set by Nix's runCommand)
    echo "Output: $out"
  '';
};

Rule: Use ${var} for Nix variables, $var for shell variables.

Default Registries

Turnkey includes defaults for known problematic crates:

Default rustcFlagsRegistry:

{
  serde_json = ["--cfg" ''fast_arithmetic=\"64\"''];
  rustix = ["--cfg" "libc" "--cfg" "linux_like" "--cfg" "linux_kernel"];
}

Default buildScriptFixups:

• serde_core - Generates out_dir/private.rs
• serde - Generates out_dir/private.rs
• ring - Compiles native crypto library (~440 lines of build commands)

User-provided values override defaults (same key) or extend them (new keys).

Best Practices

1. Check build.rs first - Read the crate's build.rs to understand what it does
2. Start simple - Try rustcFlagsRegistry before complex fixups
3. Version your fixups - Use versioned keys if build.rs changes between versions
4. Document complex fixups - Explain what the original build.rs does

Debugging Tips

Inspect Generated rules.star Files

cat .turnkey/rustdeps/vendor/serde_json@1.0.140/rules.star

Look for:

• rustc_flags - Should include cfg flags
• env - Should include OUT_DIR if fixup crate
• deps - Dependencies resolved correctly

Check Fixup Output

ls -la .turnkey/rustdeps/vendor/ring@0.17.14/out_dir/

Trace Feature Resolution

grep -A20 "rust_library" .turnkey/rustdeps/vendor/serde@*/rules.star | grep features

Creating a New Generator

1. Create CLI Tool

Create a CLI tool in cmd/newlang-gen/:

// cmd/newlang-gen/main.go
package main

import (
    "flag"
    "os"
    // ...
)

func main() {
    lockFile := flag.String("lock", "newlang.lock", "Path to lock file")
    output := flag.String("o", "", "Output file (default: stdout)")
    prefetch := flag.Bool("prefetch", true, "Fetch Nix hashes")
    flag.Parse()

    // 1. Parse lock file
    deps := parseLockFile(*lockFile)

    // 2. Prefetch packages if requested
    if *prefetch {
        prefetchHashes(deps)
    }

    // 3. Output TOML
    outputTOML(deps, *output)
}

2. Create Cell Builder

Create nix/buck2/newlang-deps-cell.nix:

{ pkgs, lib, depsFile }:

let
  deps = builtins.fromTOML (builtins.readFile depsFile);

  fetchDep = name: info:
    pkgs.fetchurl {
      url = info.url;
      hash = info.hash;
    };

  depSources = lib.mapAttrs fetchDep deps.deps;
in
pkgs.runCommand "newlang-deps-cell" {} ''
  mkdir -p $out

  # Generate cell .buckconfig
  cat > $out/.buckconfig << 'EOF'
  [cells]
      newlang-deps = .
  [buildfile]
      name = rules.star
  EOF

  # Generate rules.star for each dependency
  ${lib.concatStrings (lib.mapAttrsToList (name: src: ''
    mkdir -p $out/${name}
    cp -r ${src}/* $out/${name}/
    cat > $out/${name}/rules.star << 'EOF'
    # Generated build rules for ${name}
    newlang_library(
        name = "${lib.last (lib.splitString "/" name)}",
        srcs = glob(["*.newlang"]),
        visibility = ["PUBLIC"],
    )
    EOF
  '') depSources)}
''

3. Add to Devenv Module

Update nix/devenv/turnkey/buck2.nix to support the new language.

4. Add Configuration Options

Add options for deps file path and any language-specific configuration.

Testing

# Generate deps file
newlang-gen > newlang-deps.toml

# Verify it's valid TOML
nix eval --expr 'builtins.fromTOML (builtins.readFile ./newlang-deps.toml)'

# Build the cell
nix build .#newlang-deps-cell

# Check generated content
ls result/

Development Setup

Set up your environment to contribute to Turnkey.

Prerequisites

• Nix with flakes enabled
• direnv (recommended)
• Git

Clone and Enter Shell

git clone https://github.com/firefly-engineering/turnkey.git
cd turnkey
direnv allow  # or: nix develop

Repository Layout

turnkey/
├── flake.nix           # Main flake (self-usage example)
├── toolchain.toml      # Example toolchain config
├── nix/
│   ├── flake-parts/    # Flake-parts module
│   ├── devenv/         # Devenv module
│   ├── registry/       # Default registry
│   ├── buck2/          # Buck2 integration
│   └── packages/       # Tool packages
├── cmd/                # CLI tools (Go)
├── docs/               # Documentation
└── examples/           # Example projects

Making Changes

Nix Code

1. Edit files in nix/
2. Stage changes: git add nix/
3. Re-enter shell to test: exit && nix develop

Go Code

1. Edit files in cmd/
2. Build: tk build //src/cmd/...
3. Run: tk run //src/cmd/mytool:mytool

Documentation

1. Edit files in docs/
2. Build book: tk build //docs/user-manual:user-manual
3. Preview: tk run //docs/user-manual:user-manual

Running Tests

# All tests
tk test //...

# Specific package
tk test //src/go/pkg/syncer:syncer_test

Pre-commit Hooks

Turnkey uses pre-commit hooks for:

• Nix flake check
• Monorepo dependency check
• Rust edition check

Hooks run automatically on commit.

Code Style

Nix

Formatting

• 2 space indentation
• Multi-line function parameters
• Aligned braces

{
  config,
  pkgs,
  lib,
  ...
}:

Module Pattern

{
  options = {
    # Option definitions
  };

  config = lib.mkIf cfg.enable {
    # Implementation
  };
}

Naming

• Use descriptive attribute names
• camelCase for local variables
• kebab-case for package names

Starlark (Buck2)

Rule Definitions

def _my_rule_impl(ctx: AnalysisContext) -> list[Provider]:
    """Implementation of my_rule.

    Args:
        ctx: Analysis context from Buck2
    """
    pass

my_rule = rule(
    impl = _my_rule_impl,
    attrs = {
        "srcs": attrs.list(attrs.source()),
    },
    doc = "Short description of the rule.",
)

Naming

• snake_case for functions and variables
• PascalCase for providers
• _prefix for private functions

Go

Standard Go formatting with gofmt.

Package Comments

// Package syncer provides dependency synchronization.
package syncer

Commit Messages

Use conventional commits:

feat: add zig toolchain support
fix: correct Python deps cell generation
docs: update troubleshooting guide
refactor: simplify registry pattern

Prefix types:

• feat: - New features
• fix: - Bug fixes
• docs: - Documentation
• refactor: - Code restructuring
• test: - Test additions
• chore: - Maintenance

Testing

Running Tests

All Tests

tk test //...

Specific Packages

# Go packages
tk test //src/go/pkg/syncer:syncer_test

# Rust crates
tk test //src/rust/prefetch-cache:prefetch-cache-test

# Python modules
tk test //src/python/cargo:test_features

Test Categories

Unit Tests

Located alongside source code:

pkg/
├── syncer.go
└── syncer_test.go

Integration Tests

Located in e2e/:

e2e/
├── fixtures/
│   ├── greenfield-go/
│   └── multi-language/
└── run_e2e.sh

Nix Testing

Flake Check

nix flake check

Validates:

• Module definitions
• Package builds
• Template validity

Derivation Builds

# Build specific package
nix build .#godeps-gen

# Build prelude
nix build .#turnkey-prelude

Manual Testing

New Toolchain

1. Add to registry
2. Add to mappings
3. Add to toolchain.toml
4. Enter shell
5. Verify tk targets toolchains//...

Prelude Extension

1. Create extension files
2. Stage: git add nix/buck2/prelude-extensions/
3. Rebuild: nix build .#turnkey-prelude
4. Verify files in output

Dependency Cell

1. Generate deps file
2. Build cell: nix build .#godeps-cell (example)
3. Verify rules.star content

CI

Pre-commit hooks run:

1. nix flake check
2. monorepo-dep-check
3. rust-edition-check

All hooks must pass before commit.

Submitting Changes

Before You Start

1. Check existing issues for related work
2. For large changes, open an issue first to discuss approach
3. Fork the repository

Development Workflow

1. Create a feature branch:

   git checkout -b feat/my-feature
   

2. Make your changes

3. Test thoroughly:

   tk test //...
   nix flake check
   

4. Commit with conventional message:

   git commit -m "feat: add zig toolchain support"
   

Pull Request Process

1. Push your branch:

   git push -u origin feat/my-feature
   

2. Open a Pull Request on GitHub

3. Fill out the PR template:

   • Summary of changes
   • Test plan
   • Related issues

4. Wait for review

PR Checklist

• Tests pass (tk test //...)
• Nix flake checks pass (nix flake check)
• Pre-commit hooks pass
• Documentation updated if needed
• Commit messages follow convention

Review Process

• Maintainers will review within a few days
• Address feedback with additional commits
• Once approved, maintainer will merge

After Merge

• Delete your feature branch
• Pull latest main
• Thanks for contributing!

Getting Help

• Open an issue for questions
• Tag maintainers if stuck on review
• Check existing PRs for examples