Run simulations in CI

A working local simulation is the starting point. Check in its recipe, target adapter or SDK runner, and application dependency lockfiles. CI executes your application in your runner; Firedrill retains the batch and evidence. The workflow uses the same project/environment and scenario semantics as local execution. Add the Firedrill CI App when you want an App check and an updated PR comment; ordinary Actions and its job summary also work without it.

Generate a GitHub Actions workflow

From the initialized application repository:

firedrill ci init

Choose the saved recipe and when to run. Arrow keys select one option; Space selects checkboxes; Enter continues. The wizard asks for the branch, schedule and SDK runner only when relevant, offers a dependency install step, and shows a review before writing. If the project/environment is not selected yet, run firedrill init once first. For coding agents and CI, keep the trigger explicit. With one valid saved recipe:

firedrill ci init --trigger manual --json --non-interactive

Add the recipe path after init if more than one exists or it lives outside the current directory and firedrill/. Multiple candidates return exact paths rather than guessing. Add the application’s real --install-command when dependencies need preparing. GitHub is the default provider and the installed CLI version is pinned automatically. JSON, --non-interactive and CI never prompt. The generated file is .github/workflows/firedrill.yml. Generation writes files only: it does not contact GitHub, create secrets, invoke the target, or run a simulation. Existing differing workflow files are not overwritten. The receipt includes a repeatable noninteractive command with the resolved selections. The recipe and native runner script must be regular checked-in files in this repository. Avoid absolute local working directories in command targets. The generator can pin the exact installed CLI version automatically. An explicit --cli-version or --cli-archive remains available when you deliberately select another reviewed client.

Add the GitHub App report

Install the Firedrill CI App. GitHub asks which account and repositories to authorize. After selecting the repository, use the terminal wizard:

firedrill ci init --github-app

Choose the recipe, Pull request trigger and intended branch. Keep the protected environment and scoped credential steps. Coding agents can make these selections explicit with --trigger pull_request --branch YOUR_BRANCH --github-app and an exact recipe path when needed. The first verified workflow report connects that repository and workflow to the selected Firedrill project and environment. You do not need to connect the repository in the portal. The workflow needs id-token: write to prove its GitHub identity; it does not receive the App’s private key or a repository-write token. Firedrill publishes a check and updates one PR comment with simulation counts, pass/fail or inconclusive status, execution problems and an authenticated detailed-report link. If a runner creates several batches, their results are combined so a passing batch cannot hide an earlier failure. The check stays in progress until the workflow and required assessments finish. The generated App-enabled workflow requires successful report delivery. An unavailable App or missing installation is reported as a reporting problem, not an agent failure, and does not rerun the simulation. For reporting recovery from the same authorized workflow job:

firedrill ci report BATCH_ID --github-app --require-app

Use the saved batch ID. Reporting reads retained results; it does not execute your agent again. The signed workflow identity must still match the original run and attempt.

Select when it runs

Repeat --trigger for each desired event:

Trigger	Generated event	Additional selection
`manual`	`workflow_dispatch`	None
`push`	`push`	Target branch; local `origin/HEAD` is used when recorded, otherwise select it
`pull_request`	`pull_request`	Same target branch selection; fork PRs are skipped
`schedule`	`schedule`	Required five-field UTC `--cron`
`release`	`release`, type `published`	None

A scheduled example:

firedrill ci init firedrill/scenario.json \
  --trigger schedule --cron '0 9 * * 1' --json --non-interactive

Choose one workflow generation command for the file you intend to create. If a generated workflow already exists, review changes deliberately instead of expecting a second invocation to merge new triggers.

Protect the credential

Create a GitHub environment named firedrill-ci with required reviewers and appropriate deployment-branch restrictions. Review the exact code that will run before approval releases secrets. A same-repository branch is not inherently trusted. An authorized human issuer can create a project/environment-scoped machine credential. For a tool-free command/HTTP simulation, the typical capabilities are read, runs:write, and runs:execute:

credential_directory="$(mktemp -d)"
firedrill credentials create \
  --label github-actions \
  --capability read \
  --capability runs:write \
  --capability runs:execute \
  --expires-in-seconds 2592000 \
  --output "$credential_directory/ci.json" \
  --idempotency-key github-actions-first-credential \
  --json --non-interactive

The credential is written privately, not printed. Keep the file and adjacent issuance intent protected. Through your approved secret-management process, save the file’s token value—not the entire JSON document—as environment secret FIREDRILL_TOKEN. Do not commit either private file. Use a distinct saved key for a deliberately new issuance and the same unchanged key/input to recover uncertainty. Give the credential only additional capabilities actually required by the recipe’s Tools or actions. Machine credentials cannot issue other credentials. Your application’s model/provider credentials are separate secrets; supply them only to the approved execution step, never to recipe files or install steps.

What the workflow does

The generated workflow:

Checks out the selected revision with read-only repository permissions and no persisted Git credential.
Installs a pinned CLI version using Node.js 24.
Optionally installs application dependencies without the Firedrill token.
Waits for the protected firedrill-ci environment before the job can use its secrets.
Runs the command/HTTP recipe on the GitHub runner with --origin ci --non-interactive.
Writes a compact job summary with retained counts, a separate gate, and an authenticated report link.
Runs an always-executed reporting step that reads retained results and enforces their exit policy, including native SDK scripts that otherwise exit successfully.

For an HTTP target, prepare your application endpoint in the workflow before the simulation step. Firedrill posts the task envelope from the CI runner; it does not start or deploy that endpoint. For a command target, the runner executes literal argument values without a shell. The generated workflow excludes fork pull requests and never generates pull_request_target execution. Read pull-request CI before enabling repository PR code with credentials.

Native SDK runners

For a callback or explicitly isolated target, select the checked-in script that already calls the SDK:

TypeScript
Python

firedrill ci init firedrill/scenario.json \
  --trigger manual \
  --install-command 'npm ci' \
  --runner node --runner-arg scripts/simulate.mjs \
  --json --non-interactive

firedrill ci init firedrill/scenario.json \
  --trigger manual \
  --install-command 'python3 -m pip install -r requirements.txt' \
  --runner python3 --runner-arg scripts/simulate.py \
  --json --non-interactive

Review Python runtime setup and pin the application’s dependencies. The SDK itself does not need Node.js; this generated workflow uses Node.js for its CLI preview step.

The workflow first previews the recipe, then executes that script directly—not as a command inside another simulation. The script must read FIREDRILL_RECIPE_FILE or the same pinned recipe and bind the actual callback. The SDK saves private batch-reference receipts before target execution; the final reporting step retrieves those batches and enforces their retained policy, even if the script exits zero. Missing receipts do not become a green result. A failed script remains a failed workflow. The generator supplies FIREDRILL_SUBMISSION_ORIGIN=ci. Native clients inherit that declaration unless explicitly overridden. Use the TypeScript or Python recovery semantics; do not blindly start another simulation after uncertainty.

Understand CI results

A recipe may be observation-only, assessed, gated, or both assessed and gated. A configured gate is allow, block, or inconclusive; no gate is not_configured. Execution completion is not a quality verdict. Missing assessment is not zero, and an evaluator outage is not agent failure. The CLI exits nonzero for a blocking/inconclusive gate, failed execution, incomplete work, or an infrastructure error. In other CI systems, native runners can use the SDK’s simulationExitCode(result) in TypeScript or simulation_exit_code(result) in Python. See exit codes. Ordinal rubric thresholds use gate.minimumLevels. Fractional evaluator-score thresholds use gate.minimumScores; the two are not silently converted. For example, with an authored rubric dimension named answer_quality:

{
  "gate": {
    "requireCompleted": true,
    "requireExactChecks": true,
    "minimumScores": { "answer_quality": 3.0 }
  }
}

Use your own rubric dimension IDs and thresholds. A missing or unassessable required score is inconclusive, not a pass or a zero score. With no gate, successful execution is reported as completed, not as proof of correctness. The portal and CLI show the same retained records. --origin ci records declared submission provenance; it is not proof of target source or workflow execution. GitHub metadata and the report link help locate the actual CI job. Summary-write failure is separate from execution and gate state. Open CI runs in the portal to inspect runs in the selected project and environment, filter by repository or workflow, and follow the original PR and Actions links. Actual verified repository connections appear there when available; source metadata alone does not create a connected repository.

Other CI systems

Install the CLI and your application dependencies, select the project/environment, inject FIREDRILL_TOKEN through protected secrets, and run:

firedrill simulate firedrill/scenario.json --origin ci --json --non-interactive

Use the native SDK script instead for callbacks. Keep journals private and preserve the original operation ID after interrupted work. Use follow for observation; do not run the target again to retrieve an existing report.

Get started

Coding agents

Simulations and evidence

Synthetic Tools

Automation

Web app and access

Core concepts

Files and reference

Run simulations in CI

Generate a GitHub Actions workflow

Add the GitHub App report

Select when it runs

Protect the credential

What the workflow does

Native SDK runners

Understand CI results

Other CI systems

​Generate a GitHub Actions workflow

​Add the GitHub App report

​Select when it runs

​Protect the credential

​What the workflow does

​Native SDK runners

​Understand CI results

​Other CI systems

Generate a GitHub Actions workflow

Add the GitHub App report

Select when it runs

Protect the credential

What the workflow does

Native SDK runners

Understand CI results

Other CI systems