TypeScript SDK - Firedrill

Requires Node.js 22 or newer.

npm install @firedrill-run/sdk

Select authorized scope

Run firedrill init with the separately installed CLI; it offers sign-in if needed. Then construct your client from the public configuration:

import { Firedrill } from "@firedrill-run/sdk";

const client = Firedrill.fromConfig();

fromConfig(path?, options?) defaults to firedrill.config.json in the current directory. It reads project/environment IDs, not repository credentials, and never opens a browser. In CI, supply FIREDRILL_TOKEN from the secret store. For a file-free client, use new Firedrill({projectId, environmentId}) with the actual authorized IDs, or set FIREDRILL_PROJECT_ID and FIREDRILL_ENVIRONMENT_ID. Firedrill selects its production service automatically. Keep scope consistent across configuration, options, and environment variables; disagreement fails before a request.

Simulate your application

This example assumes your existing application exports an async runAgent(input, {signal}) function from ./src/agent.js. Adjust that import and the callback to your actual entry point; do not substitute generated answers for application execution. Save as simulate.mjs:

import { Firedrill } from "@firedrill-run/sdk";
import { runAgent } from "./src/agent.js";

const client = Firedrill.fromConfig();
const target = async (input, context) => {
  context.observe("application.invoked", { scenarioId: context.scenario.id });
  return await runAgent(input, { signal: context.signal });
};

const result = await client.simulate({
  name: "Support reply variations",
  scenarios: [
    { id: "clear-request", name: "Clear request", input: { message: "Where is my order?" } },
    { id: "missing-details", name: "Missing details", input: { message: "Please help." } },
  ],
  repetitions: 2,
  concurrency: 1,
}, target);

console.log({
  batchId: result.batch.id,
  operationId: result.operationId,
  gate: result.gate,
  reportUrl: result.reportUrl,
});

node simulate.mjs

These are four real callback invocations: two scenarios × two repetitions. The example requests observation, not grading. Add only the checks or rubric your task requires. The callback receives detached JSON input and a context containing run/attempt IDs, ID-only scenario context, configuration, cancellation signal, deadline, run-scoped Tools, observation and attachment methods. It returns JSON-compatible application output. context.observe records runner-attributed evidence, not independent proof of an external side effect. Native callbacks share your process; they are not a sandbox. Honor cancellation in the application. Start sequentially unless you have reviewed shared state and reset behavior. allowParallel: true on a fresh simulation records explicit shared-state acknowledgement; it does not create isolation.

Use a saved recipe

const result = await client.simulate("firedrill/scenario.json", target);

A callback recipe may omit target. A recipe for an existing command or HTTP endpoint supplies target and needs no callback:

const result = await client.simulate("firedrill/http-scenario.json");

Paths in command targets retain the process working directory’s meaning. They are not rebased to the recipe file. Commands receive the versioned task envelope on stdin; HTTP receives the same envelope as a POST. Read client.schema("target-invocation") for the exact contract.

Inspect and recover

Operation	Meaning
`await client.preview(recipe)`	Validate without invoking a target or reserving capacity
`await client.status(batchId)`	Read batch state
`await client.result(batchId)`	Read retained results
`await client.list(batchId, {limit: 25})`	Page retained runs
`await client.report(runId, {limit: 20})`	Read one run’s evidence page
`await client.follow(batchId, {timeoutMs: 30000})`	Observe without dispatching
`await client.run(batchId, target)`	Attach your runner to an existing batch
`await client.resume(operationId, target)`	Reconcile the original private execution journal

Use returned cursors unchanged with the same selection. Large results may use a compact result and retained read handles; omitted values are not null. Keep the operation ID and private journal after interruption. Do not call simulate again to repair an uncertain acknowledgement.

CI and assessment

Use client.simulate(recipe, target, {submissionOrigin: "ci"}) in CI, or set FIREDRILL_SUBMISSION_ORIGIN=ci. The CI generator runs your script directly. For the saved-recipe version of the script above, replace its simulation and output section with this CI-aware form. Keep the same actual application target:

const recipePath = process.env.FIREDRILL_RECIPE_FILE ?? "firedrill/scenario.json";
const result = await client.simulate(recipePath, target);
console.log(result.batch.id, result.reportUrl, result.gate);

const counts = result.batch.counts;
const pendingAssessment = result.assessmentCounts
  ? (result.assessmentCounts.pending ?? 0) > 0
  : result.runs.some(run => run.assessment === "pending");
const unfinished = pendingAssessment ||
  ["interrupted", "cancelled", "running", "queued"].some(key => counts[key] > 0);
process.exitCode = ["block", "inconclusive"].includes(result.gate)
  ? 1 : unfinished ? 6 : counts.error > 0 ? 5 : 0;

The generated workflow supplies CI origin and protected credentials. This script reads the selected recipe and propagates its retained gate/execution state instead of treating a printed report as a passed job. The canonical result.gate is not_configured, allow, block, or inconclusive. Read execution, evidence coverage, and assessment separately. Fractional semantic provider scores are distinct from ordinal rubric levels; missing results are not zero and scores are not probabilities. An evaluator problem does not turn a completed target into a failed agent. See errors and recovery before implementing retries.

​Select authorized scope

​Simulate your application

​Use a saved recipe

​Inspect and recover

​CI and assessment

Select authorized scope

Simulate your application

Use a saved recipe

Inspect and recover

CI and assessment