Skip to main content
This guide explains Firedrill’s repository contract. The files belong beside the agent code, can be reviewed in pull requests, and work without an account. The npm packages are still an unpublished release candidate. After an authorized publication, the normal project-local installation will be npm install --save-dev @firedrill/cli, followed by npx firedrill init. Until then, use a reviewed packed release artifact or an authorized framework source checkout. The exact available flags are documented in the CLI reference. For your existing agent, start with local tools:
Choose or create tools, inspect their data and behavior, then connect the existing agent. No scenario or test is required to get a usable backend. To learn the complete testing loop through an explicit example instead:
That creates repository-owned source, validates it, runs one passing drill in an isolated world, prints the self-contained report path, and opens the local inspector. No account or network service is involved. For saved results, open .firedrill/reports/index.html. This central page lets you search and filter recorded executions, then open an individual report’s task, checks, tool calls and data changes. It refreshes when the CLI or SDK saves reports; no inspector server is required to browse it. A custom --report-dir has its own index.html. The CLI prints this path once as All reports, and JSON/SDK results expose reportIndex. Individual run folders remain portable evidence bundles. The central index is just navigation over verified bundles; unavailable reports are listed without unsafe links, and bounded-history limits are stated on the page. Copy the index and its run folders together if you intend to share a browsable collection. Keep generated output out of Git.

​
1. Pick a verified starting path

In an interactive terminal, firedrill init guides tool selection, optional authoring, and local startup. A catalog package requires installation consent if missing; custom tools need no key or account. The wizard does not require you to invent a test task before starting the backend. In JSON, CI, or piped use, bare init remains read-only. Use --tool, --custom, or a legacy --path to make an explicit setup selection; --start opts into foreground execution.
The explicit --path firedrill-agent and --path coding-agent forms install the same skill and repository brief under .agents/. Firedrill Agent uses the separately installed @firedrill/agent and your ANTHROPIC_API_KEY; its default task is preparing tools, not inventing a test suite. The template path remains an explicit complete runnable example. The legacy manual path creates a world shell and probe tool; prefer --custom <id> for an editable stateful backend. Existing files are never silently replaced. Setup ensures .firedrill/ is Git-ignored. Alternatively, copy examples/quickstart into a temporary directory or inspect it in place. It contains one intentionally plain agent and one Tool so the framework concepts stay visible.
agent.mjs is the agent being tested; the firedrill/ folder describes its controlled surroundings and tests. The Tool declaration owns record/input/output schemas; its behavior module owns the fake consequences. world.yaml owns shared starting conditions and permissions, scenarios provide starting data or variations, and each drill owns its task and assertions. The example agent is a deterministic HTTP client, not an LLM. The generated template follows the same folders, with its demonstration agent kept separately in firedrill-example/agent.mjs. Both are teaching conventions, not required project layouts. Resource files are discovered recursively under sourceRoot; descriptive kebab-case filenames with .tool, .scenario, .target, .drill or .suite suffixes make their purpose obvious. YAML, YML and JSON are supported. References use stable in-file IDs, so changing a display title need not move a file. Existing flat projects remain valid and need no migration. Do not rerun init just to rearrange an existing world. If you choose to move source, preserve IDs, update relative Tool module paths and any configured world path, then validate. Set sourceRoot and world in firedrill.json if you prefer another source folder or world filename; neither tools/ nor any other example subfolder is mandatory. Run firedrill validate, inspect the discovered work with firedrill plan, then run firedrill. A fresh SQLite world is created, the agent is invoked with only its declared binding, assertions inspect the resulting consequences, and the terminal prints a local HTML report path.

​
What belongs in Git

The ignored side is reproducible runtime output and may contain synthetic data, model output, and evidence. Provider keys belong in the agent’s normal ignored environment files or secret manager—never in either Firedrill source or reports. Repository data is the reproducible starting definition, not a live database dump. Each trial, retry, and concurrent run materializes its own SQLite file from the pinned build plus scenario. Runtime Tool calls mutate that isolated file and never write records back into YAML or JSON. Editing repository data creates the starting state for later builds and runs; retained older worlds and evidence remain unchanged. All Tools selected for one trial share that trial’s world database, which is what makes cross-Tool consequences, the clock, pending events, and the ordered journal atomic. A project may therefore retain many SQLite worlds without assigning one database per vendor or Tool.

​
2. Replace the fixture with the agent’s real boundaries

Work from the interfaces the agent already uses:
  • Model each required action surface as a Tool operation with typed input, output, and deterministic behavior.
  • Put baseline records, actors, permissions, time, and initial events in the world.
  • Put each meaningful starting condition or provider failure in a scenario.
  • Choose one target matching how the agent already runs: module, command, local HTTP, or an SDK callback.
  • Give the target only the direct, HTTP, MCP, or CLI bindings it needs.
  • Write drills around observable consequences and safety invariants, not phrasing in the model response.
Tools are not limited to REST APIs. They describe capabilities and consequences; direct, HTTP, MCP, and CLI are transport adapters over the same Tool behavior and SQLite world. The generic HTTP adapter does not automatically reproduce a vendor’s URL and payload conventions. Keep that translation at the agent’s normal client seam or in a small repository adapter instead of adding vendor branches to Firedrill core.

​
Reuse an installed Tool package when one fits

A reusable Tool is an ordinary package dependency, not a framework feature switch. Install it with the project’s package manager, then select it once in firedrill.json:
Install the selected Tool package itself; do not add its Firedrill implementation dependencies to the application. The compiler embeds the approved Tool behavior runtime into the locked artifact. The generated Tool catalog shows operation-level fidelity for known packs. The selected package’s own documentation gives its Tool id, operations, state contract, and setup. firedrill validate reads and locks the selected declaration without executing behavior. Use firedrill tool inspect <tool-id> to see exactly what was selected, then firedrill tool validate <tool-id> or run a drill to execute it locally. Your repository still owns its initial data, actors, scenarios, targets, and drills. Firedrill never edits the installed package. Browse the reviewed community catalog with firedrill tool list, or find one integration with firedrill tool search <text>. The printed install source is exact and can be passed to firedrill tool add <source> --install after review.

​
3. Keep the agent integration at one seam

For a command target, Firedrill sends a JSON invocation on stdin and supplies the declared binding variables, such as FIREDRILL_HTTP_URL and FIREDRILL_HTTP_TOKEN, their MCP equivalents, or FIREDRILL_CLI_URL and FIREDRILL_CLI_TOKEN. For an external target, runDrills() supplies the same values in binding.environment. A direct binding is available only to a module or callback target that declares it. Firedrill does not become the model-provider credential store. An external callback uses the provider configuration already available to its owning process. A command target can map only the host variables it needs:
Unlisted host variables are not inherited by the command. The target’s timeoutMs covers the complete agent interaction, including all model turns and Tool calls, so choose it for the slowest expected end-to-end loop rather than one request. Repoint the agent’s existing test configuration, or use a separate test-only adapter around its ordinary entry point. Do not modify production agent logic, duplicate every action, or scatter test-mode branches through business logic. When an ordinary test needs different starting data or a temporary Tool behavior, use the repository-level SDK rather than editing source and restoring it:
The setup object is serialized, normalized, hashed, and compiled into a derived immutable build. It may add or replace starting state rows and actors, set virtual time, activate declared faults, append initial events, select installed Tool packages, point one declared Tool at a repository-owned behavior module, and map temporary bindings onto names the agent already understands. It never writes those choices back to YAML, JSON, or SQLite behind the report. Command targets receive binding aliases automatically; caller-owned targets pass binding.environment through the agent’s existing configuration seam. Unsupported protocol mappings fail before agent execution. Full details are in the @firedrill/sdk guide. HTTP-bound agents can discover loaded operations at GET $FIREDRILL_HTTP_URL/v1/tools and call one at POST /v1/operations/{packageId}/{operationId} with bearer authentication. Discovery is not an access grant: the kernel checks actor permissions on invocation. MCP-bound agents use the supplied Streamable HTTP URL and token; discovered names are {packageId}.{operationId}. CLI-bound agents call firedrill world tools --json and firedrill world call <tool-id> <operation-id> --input '{...}' --json. The protocol READMEs define the exact envelopes.

​
4. Run and diagnose

Exit code 0 means every selected drill passed. Exit code 1 means source, execution, or assertions failed. Exit code 2 means the CLI invocation itself was invalid. Human and JSON modes carry the same diagnostics and report locations. Every trial retains its exact world and writes terminal, JSON, JSONL, JUnit, and self-contained HTML evidence under the current project’s .firedrill/ directory. That directory is generated and Git-ignored by default. Use firedrill report verify <report-directory> to check the exact file set, hashes, schemas, identities, evidence ordering, and generated projections without an account or network. This proves bundle integrity, not authorship. The report’s reproduction command loads its content-addressed build with --build-hash and reruns the recorded seed. Keep that project-local build directory while reproducing; if it was removed, restore the source revision and setup that produced the displayed hash and compile it again. A drill timeline can span hours of virtual time while running locally in minutes. It declares actors, ordered interactions, a horizon, invariant checkpoints, and Tool-call and event budgets; it is still a drill and uses the same runner and evidence. Watch mode queues edits and reruns without overlapping. To inspect change, compare two verified report directories:
Read the compatibility grade before interpreting deltas. A build or Tool-lock change is descriptive evidence, not proof that the agent regressed. Tool conformance uses ordinary drills rather than a second test language. Name the suite <tool-id>-conformance (or pass --suite), cover every declared operation/error/event/fault/subscription, and run firedrill tool test <tool-id>. Firedrill executes it twice against one immutable build and checks same-seed state and trajectory hashes. After it passes, a human who owns the source may prepare a non-uploading review bundle:
This copies only the Tool declaration and its exact local behavior dependency closure, blocks common secret patterns, writes checksums and a portable conformance summary, and never overwrites, uploads, or opens a pull request.

​
5. Let an authoring agent iterate to green

A coding agent should begin with .agents/firedrill/BRIEF.md and .agents/skills/firedrill/SKILL.md, inspect the real agent’s tool clients, existing mocks, fixtures, and failure tests, create or select Tool packages, then run firedrill validate --json repeatedly until diagnostics are empty. It should run a small passing and intentionally failing drill before adding breadth. It must not invent unsupported fidelity or change production behavior merely to satisfy a fixture. The optional local Firedrill Agent follows that same skill rather than a private format:
It uses the Claude Agent SDK with the developer’s key. It may send selected repository content to Anthropic under Anthropic’s applicable terms, but sends nothing to Firedrill Cloud. Repository discovery/search skips known secret and generated paths; file edits stay inside the selected repository; shell, generic web, Git publication, and subagents are disabled. A run defaults to 40 turns, a $2 spend ceiling, and a 15-minute deadline. These controls reduce accidental exposure but do not sandbox ordinary repository code that the Agent authors and later executes. Review its diff as you would any coding-agent change. The compiler and drill runner—not the Agent’s narrative—remain the authority. Published JSON Schemas are available from @firedrill/compiler/schema/* and @firedrill/contracts/schema/*. Source diagnostics include stable codes, file locations, paths, and corrective suggestions for machine use.