Scenarios

A scenario describes one situation your agent should handle. It names the personas who take part, the evaluations that score the result, events that happen along the way and when the conversation stops.

Example

scenario.json

{
  "schema": "firedrill.native-scenario.v1",
  "name": "Refund for a damaged order",
  "description": "",
  "instruction": "Ask for a refund for a damaged item. Escalate if the agent asks for a photo twice.",
  "personas": [
    { "origin": "native_library", "kind": "persona", "id": "<persona version id>", "revision": 1, "digest": "<digest>" }
  ],
  "evaluations": [
    { "origin": "native_library", "kind": "evaluation", "id": "<evaluation version id>", "revision": 1, "digest": "<digest>" }
  ],
  "conditions": [],
  "events": [],
  "max_turns": 8,
  "stop_rules": ["Stop once a refund is confirmed or clearly refused."]
}

Save the persona and evaluation first. Their version ID, revision and digest come from the library save output or from firedrill library show <version-id>.

firedrill library save --kind scenario --file scenario.json

How a scenario runs

Turns. One completed answer from your agent is one turn.
Instruction. The simulated user follows it, together with the persona.
Personas. Each repetition uses the next persona in the list, in order.
Stop rules. The simulated user decides when they are met and ends the conversation as finished or as giving up.
Turn limit. When max_turns is reached, the drill ends with turn_limit.
Conditions. Notes for people. They are not sent to the simulated user or checked.

Your agent never sees the instruction, the persona’s goals or the evaluations.

Fields

Field	Description
`schema`	`firedrill.native-scenario.v1`, `v2` or `v3`.
`name`, `description`	For people reading it.
`instruction`	What the simulated user should do. Required.
`personas`	References to persona versions. Needed for conversations.
`evaluations`	References to evaluation versions. At least one is needed to run.
`conditions`	Notes only.
`events`	Things that happen during the drill. See below.
`max_turns`	1 to 1000.
`stop_rules`	When the simulated user should end the conversation.

A reference points at one exact version: {"origin": "native_library", "kind", "id", "revision", "digest"}.

Events

Events change the drill at a turn boundary.

`kind`	What happens	`dependency`	`message`
`user_message`	The user sends this exact message.	`null`	The text
`dependency_unavailable`	A synthetic tool starts failing.	The dependency alias	`null`
`dependency_restored`	The tool works again.	The dependency alias	`null`

"events": [
  { "kind": "dependency_unavailable", "after_turn": 2, "dependency": "payments", "message": null },
  { "kind": "dependency_restored", "after_turn": 4, "dependency": "payments", "message": null }
]

While a dependency is unavailable, your agent’s calls to it return HTTP 503 with dependency_unavailable. Use this to see how your agent handles an outage.

Timed events (v3)

firedrill.native-scenario.v3 events can run after a delay instead of a turn. Set exactly one of after_turn or after_active_ms, plus occurrences and repeat_every_ms to repeat an event.

{ "kind": "user_message", "after_turn": null, "after_active_ms": 60000,
  "repeat_every_ms": null, "occurrences": 1, "dependency": null,
  "message": "Any update on my refund?" }

Timing uses real active time. It is never sped up. v3 also adds evolution. It decides what the simulated world does if your machine stops running the drill, for example when the runner disconnects:

{"mode": "pause_with_agent", "continuation_seconds": 0} pauses timed events with your agent.
{"mode": "world_only", "continuation_seconds": 600} keeps the simulated world running in Firedrill for up to that many seconds. The plan review asks you to approve this.

Task and event inputs

Not every agent is a chat. With firedrill.native-scenario.v2 or v3, target_input replaces the conversation with one input:

Task

"target_input": { "kind": "task", "input": { "ticket_id": "T-1042", "priority": "high" } }

Webhook event

"target_input": {
  "kind": "event",
  "method": "POST",
  "path": "/orders",
  "headers": { "content-type": "application/json" },
  "body_base64": "<base64 of a JSON document>",
  "signature_header": "x-firedrill-signature"
}

Use null for a normal conversation. An event input needs the signed-http-event.v1 driver and cannot be combined with user_message events.

Evaluations

Score the result.

Worlds

Run scenarios with synthetic tools.

Get started

Coding agents

Connect your agent

Build a world

Run drills

Read results

Account

Reference