Skip to main content
Pull-request CI executes application code from a branch. Protect the Firedrill credential and the application’s model keys before enabling it; a simulation does not make untrusted code safe.

​
Begin with a working recipe

Run the real application locally through the CLI or native SDK, inspect its retained evidence, then check in the recipe and adapter. Keep scenario intent, repetitions, dependency versions, and any rubric/gate explicit. Generate the workflow from the initialized repository:
Choose the saved simulation, select Pull requests, and review the target branch and dependency install step. Arrow keys select one option; Space selects checkboxes; Enter continues. Choose PR checks/comments if you want the Firedrill CI App report. The wizard shows a final review before writing. For coding agents or automation, use explicit selections without prompts:
Use your repository’s actual target branch. Add --install-command when application dependencies need preparing. Native callbacks need the checked-in SDK script; the terminal wizard asks for it, or use --runner and --runner-arg in automation. See native CI runners. This writes .github/workflows/firedrill.yml without executing the target or creating secrets. It does not overwrite a differing existing workflow. Install the Firedrill CI App on that repository to enable its check and PR comment. The CLI pins its own exact installed version unless you select a different reviewed version explicitly. Omit --github-app if you only want the Actions workflow and job summary.

​
Require review before credentials

Configure the generated firedrill-ci GitHub environment with required reviewers and suitable branch restrictions. Add a scoped, expiring environment secret FIREDRILL_TOKEN using the credential instructions. Keep target-model keys separate and expose them only to the approved execution step. The generated job:
  • Skips fork pull requests.
  • Uses pull_request, not pull_request_target.
  • Uses read-only repository contents, requests a signed workflow identity for App reporting, and does not retain the checkout credential.
  • Runs the checked-out application on the CI runner only after environment approval.
  • Does not automatically cancel an earlier simulation merely because a newer push arrives.
Review same-repository PRs too: a contributor can alter application code, install scripts, the recipe, or the workflow. Required review must cover the exact code and revision that will receive secrets. Do not remove the fork guard or widen event permissions just to make a check appear.

​
Inspect the retained result

The workflow records declared ci origin and available GitHub source context, including repository, commit, workflow run, attempt, job, event, and optional PR number. These are caller declarations, not signed source attestations. Its job summary links to the authenticated Firedrill report. With the App enabled, the PR also gets a simulation check and one updated comment with passed, failed, inconclusive and ungated counts. Multiple reported batches are combined; workflow execution failures cannot be hidden by a passing simulation batch. Inputs, outputs, evidence contents and credentials are not copied into the comment. Open the detailed report to see scenario results, exact checks, quality scores and captured evidence. Links retain the correct organization, project and environment, and require sign-in. CI runs in the portal also shows the PR/workflow context and actual connected-repository details when available. The App verifies GitHub’s signed workflow identity and repository/run information before posting. Caller-declared source metadata by itself cannot authorize a check or comment on another repository. Fork PRs and privileged pull_request_target handoffs are not eligible for this connection path. A completed target is not automatically correct. Semantic rubrics assess meaning and allow valid output variation; exact checks enforce precise constraints. Missing evidence stays unknown. An unavailable evaluator may make a required gate inconclusive rather than establish an agent defect.

​
Compare a baseline and candidate

Use real retained batch IDs from like-for-like populations:
Keep scenario/data revisions, rubric/evaluator versions, target configuration, and repetition counts comparable. Read missing pairs and unassessed counts rather than averaging them away. Follow returned paired-case cursors to inspect the complete selection. A descriptive comparison does not establish statistical significance or a universal model winner.

​
Recover without duplicating execution

An interrupted CI observer is not proof that its target stopped. Preserve the private journal and original operation ID, then inspect and reconcile that work before dispatching another target. firedrill follow BATCH_ID only observes; simulate --resume OPERATION_ID can continue unfinished execution. A fresh hosted runner does not contain another machine’s journal. Restore it only through an approved private artifact/state mechanism, or inspect the retained batch and use the reported recovery direction. Never commit journals or recover by changing the original scope, identity, or key. The final workflow reporting step enforces the retained result, including blocked or inconclusive SDK results even when a script exits zero. A failed script remains a failed job. The separate Firedrill gate remains visible in its report. App publication failures are reporting problems; recovering delivery does not execute the target again.