Evaluations

An evaluation is a scoring rule. You write criteria in plain language, choose the evidence to judge them on and set a threshold. Each evaluation in a scenario becomes one result in the drill’s report.

Example

evaluation.json

{
  "schema": "firedrill.native-evaluation.v1",
  "name": "Handles the refund correctly",
  "description": "",
  "method": "rubric",
  "criteria": [
    "The agent confirms the order number before acting.",
    "The agent explains the refund policy in one or two sentences.",
    "The agent does not ask for a photo more than once."
  ],
  "evidence": ["conversation", "actions"],
  "threshold": 75
}

firedrill library save --kind evaluation --file evaluation.json

How scoring works

Firedrill turns your criteria into a numbered rubric.
It rates the drill’s evidence on five levels, from “does not satisfy” to “fully satisfies”.
The level becomes a score of 0, 25, 50, 75 or 100.
The evaluation is met when the score is at or above threshold.

Correct answers can be worded differently. The rubric judges meaning, not exact text. If the evidence an evaluation needs was not recorded, the result is not assessable. It is never counted as zero or as a failure of your agent. Results from different evaluations are never averaged into one number. Each criterion keeps its own result.

Evidence

Evidence	What it contains	Can be scored
`conversation`	Messages between the simulated user and your agent	Yes
`actions`	What your agent did, including tool calls	Yes
`state_changes`	Changes in synthetic tools, before and after	Yes
`screenshots`	Browser captures from `playwright.v1`	Not yet
`recordings`	Recorded sessions	Not yet
`metrics`	Measured values	Not yet

An evaluation that needs evidence which cannot be scored yet shows as not assessable.

Fields

Field	Description
`schema`	`firedrill.native-evaluation.v1` or `v2`.
`name`, `description`	For people reading the report.
`method`	Always `rubric`.
`criteria`	1 to 32 plain-language statements.
`evidence`	At least one evidence type from the table above.
`threshold`	0 to 100.

Exact checks and focused scoring (v2)

firedrill.native-evaluation.v2 adds these required fields:

Field	Description
`dimension`	What the evaluation measures: `accuracy`, `completion`, `memory`, `grounding`, `instruction_adherence`, `policy`, `style` or `custom`.
`memory`	`null`, or a memory plan that checks whether the agent remembers facts across turns. Needs `conversation` evidence.
`style`	`null`, or `{"scope": "assistant_prose", "rules": ["em_dash"]}` to check your agent’s writing style.
`company_policies`	Policy documents the evaluator uses. Required when `dimension` is `policy`.
`exact_checks`	Counted checks with a fixed answer. See below.

state_records is optional. It compares a synthetic tool’s records before and after the drill, and needs state_changes evidence.

Exact checks

An exact check counts something and compares it with a number. It passes or fails without a model.

"exact_checks": [
  {
    "key": "one_refund",
    "source": "hosted_tool_calls",
    "service": "stripe",
    "operation": "refunds.create",
    "comparison": "eq",
    "expected": 1
  }
]

Field	Description
`source`	`hosted_tool_calls` counts calls to a synthetic tool. `style_em_dash` counts em dashes in your agent’s replies and needs `style`.
`service`, `operation`	The tool call to count. `null` for `style_em_dash`.
`comparison`	`eq`, `gte` or `lte`.
`expected`	A whole number.

Memory probes

A memory probe checks that your agent remembers something it was told earlier.

"memory": {
  "scope": "single_conversation",
  "probes": [
    {
      "key": "order_number",
      "kind": "initial",
      "expectation": "The agent still uses order 4821 without asking again.",
      "exposure_turns": [1],
      "checkpoint_turn": 4
    }
  ]
}

kind is initial, accumulated, updated or approved_forgetting. The checkpoint must come after every exposure.

Write good criteria

One behavior per criterion.
Say what good looks like: “Confirms the order number”, not “Is helpful”.
Keep tool-call counts in exact checks, not in criteria.
Start with a threshold of 75 and adjust it after reading a few reports.

Worlds

Put scenarios and tools together.

Read reports

See how results appear.

Get started

Coding agents

Connect your agent

Build a world

Run drills

Read results

Account

Reference