> ## Documentation Index
> Fetch the complete documentation index at: https://docs.firedrill.run/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations

> An evaluation scores a drill against your criteria, using the conversation, actions and state changes it recorded.

An **evaluation** is a scoring rule. You write criteria in plain language, choose the evidence to judge them on and set a threshold. Each evaluation in a scenario becomes one result in the drill's report.

## Example

```json evaluation.json theme={null}
{
  "schema": "firedrill.native-evaluation.v1",
  "name": "Handles the refund correctly",
  "description": "",
  "method": "rubric",
  "criteria": [
    "The agent confirms the order number before acting.",
    "The agent explains the refund policy in one or two sentences.",
    "The agent does not ask for a photo more than once."
  ],
  "evidence": ["conversation", "actions"],
  "threshold": 75
}
```

```bash theme={null}
firedrill library save --kind evaluation --file evaluation.json
```

## How scoring works

1. Firedrill turns your criteria into a numbered rubric.
2. It rates the drill's evidence on five levels, from "does not satisfy" to "fully satisfies".
3. The level becomes a score of 0, 25, 50, 75 or 100.
4. The evaluation is met when the score is at or above `threshold`.

Correct answers can be worded differently. The rubric judges meaning, not exact text.

If the evidence an evaluation needs was not recorded, the result is **not assessable**. It is never counted as zero or as a failure of your agent.

Results from different evaluations are never averaged into one number. Each criterion keeps its own result.

## Evidence

| Evidence | What it contains | Can be scored |
| - | - | - |
| `conversation` | Messages between the simulated user and your agent | Yes |
| `actions` | What your agent did, including tool calls | Yes |
| `state_changes` | Changes in synthetic tools, before and after | Yes |
| `screenshots` | Browser captures from `playwright.v1` | Not yet |
| `recordings` | Recorded sessions | Not yet |
| `metrics` | Measured values | Not yet |

An evaluation that needs evidence which cannot be scored yet shows as not assessable.

## Fields

| Field | Description |
| - | - |
| `schema` | `firedrill.native-evaluation.v1` or `v2`. |
| `name`, `description` | For people reading the report. |
| `method` | Always `rubric`. |
| `criteria` | 1 to 32 plain-language statements. |
| `evidence` | At least one evidence type from the table above. |
| `threshold` | 0 to 100. |

## Exact checks and focused scoring (v2)

`firedrill.native-evaluation.v2` adds these required fields:

| Field | Description |
| - | - |
| `dimension` | What the evaluation measures: `accuracy`, `completion`, `memory`, `grounding`, `instruction_adherence`, `policy`, `style` or `custom`. |
| `memory` | `null`, or a memory plan that checks whether the agent remembers facts across turns. Needs `conversation` evidence. |
| `style` | `null`, or `{"scope": "assistant_prose", "rules": ["em_dash"]}` to check your agent's writing style. |
| `company_policies` | Policy documents the evaluator uses. Required when `dimension` is `policy`. |
| `exact_checks` | Counted checks with a fixed answer. See below. |

`state_records` is optional. It compares a synthetic tool's records before and after the drill, and needs `state_changes` evidence.

### Exact checks

An exact check counts something and compares it with a number. It passes or fails without a model.

```json theme={null}
"exact_checks": [
  {
    "key": "one_refund",
    "source": "hosted_tool_calls",
    "service": "stripe",
    "operation": "refunds.create",
    "comparison": "eq",
    "expected": 1
  }
]
```

| Field | Description |
| - | - |
| `source` | `hosted_tool_calls` counts calls to a synthetic tool. `style_em_dash` counts em dashes in your agent's replies and needs `style`. |
| `service`, `operation` | The tool call to count. `null` for `style_em_dash`. |
| `comparison` | `eq`, `gte` or `lte`. |
| `expected` | A whole number. |

### Memory probes

A memory probe checks that your agent remembers something it was told earlier.

```json theme={null}
"memory": {
  "scope": "single_conversation",
  "probes": [
    {
      "key": "order_number",
      "kind": "initial",
      "expectation": "The agent still uses order 4821 without asking again.",
      "exposure_turns": [1],
      "checkpoint_turn": 4
    }
  ]
}
```

`kind` is `initial`, `accumulated`, `updated` or `approved_forgetting`. The checkpoint must come after every exposure.

## Write good criteria

* One behavior per criterion.
* Say what good looks like: "Confirms the order number", not "Is helpful".
* Keep tool-call counts in exact checks, not in criteria.
* Start with a threshold of 75 and adjust it after reading a few reports.

> [card] [Worlds](https://docs.firedrill.run/build/worlds.md)
>
> Put scenarios and tools together.

> [card] [Read reports](https://docs.firedrill.run/results/reports.md)
>
> See how results appear.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.