Python SDK - Firedrill

Requires Python 3.11 or newer. The SDK is native Python; it does not invoke the Node.js CLI.

python -m pip install firedrill-run

Authorize and select scope

If you have already run firedrill init with the optional CLI, reuse its public configuration and protected login. Interactive initialization offers sign-in if needed:

from firedrill import Firedrill

client = Firedrill.from_config()

For CI or an already authorized Python-only setup, supply a scoped FIREDRILL_TOKEN from your secret store and select the actual FIREDRILL_PROJECT_ID / FIREDRILL_ENVIRONMENT_ID pair. Then use Firedrill(), or pass project_id and environment_id explicitly. Do not put tokens in source or recipes. Run this deliberately in a human’s interactive terminal, not CI or captured coding-agent output:

import os
from firedrill import FiredrillLogin

if "FIREDRILL_TOKEN" in os.environ:
    raise SystemExit("An explicit token is selected; do not replace it with human login.")

def verify(value):
    print("Open:", value["verificationUrl"])
    print("Approval code:", value["userCode"])

def choose_organization(value):
    active = [item for item in value["organizations"] if item["status"] == "active"]
    if active:
        for item in active:
            print(item["organizationId"], item["name"])
        selected = input("Choose an organization ID shown above: ").strip()
        if selected in {item["organizationId"] for item in active}:
            return {"organizationId": selected}
    elif value["canCreate"]:
        name = input("Organization name: ").strip()
        if name:
            return {"name": name}
    raise ValueError("Finish the intended organization's authorization before continuing.")

status = FiredrillLogin().login(
    on_verification=verify,
    on_organization=choose_organization,
)
print("Authorized:", status["signedIn"])

This displays human verification details, never access or refresh tokens. Ordinary simulation methods never open a browser. To discover scope after authorization:

from firedrill import Firedrill

account = Firedrill()
for project in account.projects()["items"]:
    print(project["id"], project["name"], project["environment"]["id"])

Select an actual pair above. If you intend to create a project, account.create_project("My application", idempotency_key="my-application-project") atomically returns it and its Default environment. Save and reuse that key/name after uncertain delivery.

Simulate your application

This example assumes your application exposes run_agent in app.agent. Adapt the import and callback to the real application. Save as simulate.py:

from firedrill import Firedrill
from app.agent import run_agent

client = Firedrill.from_config()

def target(value, context):
    context.raise_if_cancelled()
    context.observe("application.invoked", {"scenarioId": context.scenario["id"]})
    return run_agent(value)

result = client.simulate({
    "name": "Support reply variations",
    "scenarios": [
        {"id": "clear-request", "name": "Clear request",
         "input": {"message": "Where is my order?"}},
        {"id": "missing-details", "name": "Missing details",
         "input": {"message": "Please help."}},
    ],
    "repetitions": 2,
    "concurrency": 1,
}, target)

print({
    "batchId": result.batch["id"],
    "operationId": result.operation_id,
    "gate": result.gate,
    "reportUrl": result.report_url,
})

python simulate.py

For Python-only scope selected through environment variables instead of a config file, change the client line to client = Firedrill(). The example invokes the callback four times and requests no grading. Replace the scenario input with tasks your actual application understands. Callbacks return JSON-compatible output. Context includes detached task input/configuration, ID-only scenario context, run/attempt IDs, cancellation/deadline helpers, run-scoped Tools, observations, metrics, and attachments. Observations remain runner-attributed evidence. Native execution is not a sandbox; timeouts do not guarantee arbitrary application work stopped.

Async applications

Use AsyncFiredrill in your existing event loop:

from firedrill import AsyncFiredrill
from app.agent import run_agent_async

client = AsyncFiredrill.from_config()

async def target(value, context):
    context.raise_if_cancelled()
    return await run_agent_async(value)

result = await client.simulate("firedrill/scenario.json", target)
print(result.report_url)

The synchronous and asynchronous clients expose the same operation names.

simulate(recipe, target=None, *, operation_id=None, allow_parallel=False, submission_origin=None, ci_context=None, on_progress=None)

accepts a recipe dictionary or selected JSON file.

Retained reads and recovery

Operation	Meaning
`client.preview(recipe)`	Validate without executing or reserving capacity
`client.result(batch_id)`	Read retained results
`client.report(run_id, limit=20)`	Read a run’s evidence page
`client.follow(batch_id, timeout=30)`	Observe without dispatching work
`client.run(batch_id, target)`	Attach a runner to an existing batch
`client.resume(operation_id, target)`	Reconcile the original private execution journal

Async callers await these methods. Preserve result.operation_id and the private journal after a partial result. result.to_dict() uses canonical wire names such as operationId and reportUrl. A retained read does not create an execution operation ID.

CI and assessment

Use submission_origin="ci" on a fresh simulation or set FIREDRILL_SUBMISSION_ORIGIN=ci. The CI generator can execute your Python script directly. For the saved-recipe version of the synchronous script above, keep the actual application target and replace its simulation/output section with:

import os
import sys

recipe_path = os.environ.get("FIREDRILL_RECIPE_FILE", "firedrill/scenario.json")
result = client.simulate(recipe_path, target)
print(result.batch_id, result.report_url, result.gate)

counts = result.batch["counts"]
pending_assessment = (
    result.assessment_counts.get("pending", 0) > 0
    if result.assessment_counts is not None
    else any(run["assessment"] == "pending" for run in result.runs)
)
unfinished = pending_assessment or any(
    counts[key] > 0 for key in ("interrupted", "cancelled", "running", "queued")
)
sys.exit(
    1 if result.gate in ("block", "inconclusive")
    else 6 if unfinished else 5 if counts["error"] > 0 else 0
)

The generated workflow supplies CI origin and protected credentials. This script propagates retained gate/execution state; a printed report alone is not a passed job. The result separates execution, observed outcome, optional assessment, and the optional gate. Valid wording can earn partial or full semantic credit without exact matching. Fractional provider scores remain distinct from rubric levels. Missing assessment is not zero; evaluation errors are not agent failures. Keep application/model credentials separate from Firedrill authorization. See errors and recovery for safe reconciliation.

​Authorize and select scope

​Human sign-in without the CLI

​Simulate your application

​Async applications

​Retained reads and recovery

​CI and assessment

Authorize and select scope

Human sign-in without the CLI

Simulate your application

Async applications

Retained reads and recovery

CI and assessment