> ## Documentation Index
> Fetch the complete documentation index at: https://docs.firedrill.run/llms.txt
> Use this file to discover all available pages before exploring further.

# Score meaning and enforce exact requirements

> Write task-specific anchors, keep fractional AI scores and hard constraints separate, and preserve unknowns.

Add evaluation only when it helps answer your question. A simulation can remain observation-only. For assessed runs, use semantic rubrics for meaning and exact checks for obligations that must hold precisely.

Your coding agent authors the task and criteria. Firedrill's Jev evaluator assesses retained text evidence; it does not execute your target or its Tools. The target's model and credentials remain in your own runner.

## Write meaningful anchors

Add a rubric to a scenario used with your actual language-producing application. This complete scenario asks for a factual delivery update without prescribing one sentence:

```json theme={null}
{
  "id": "delivery-update",
  "name": "Explain a changed delivery date",
  "input": {
    "task": "Write a concise update using the supplied facts. Do not invent a reason.",
    "facts": {
      "previousDeliveryDay": "Tuesday",
      "currentDeliveryDay": "Thursday",
      "reasonKnown": false
    }
  },
  "rubric": {
    "id": "delivery-update-quality",
    "version": "1",
    "dimensions": [
      {
        "id": "accuracy",
        "description": "Convey the changed delivery date accurately without inventing a reason.",
        "anchors": [
          "The update contradicts the delivery facts or invents a definite reason",
          "It mentions a delivery change but gets the current day wrong",
          "It gives Thursday but omits that the delivery changed",
          "It accurately explains the change to Thursday with a minor ambiguity",
          "It clearly conveys the change from Tuesday to Thursday without unsupported claims"
        ]
      }
    ]
  }
}
```

Place the object in your recipe's scenarios array, using your application's real command, HTTP endpoint, or [native callback](/sdk/typescript). The deterministic stock adapter in the first-simulation guide does not implement this language task.

The five anchors define positions 0–4. Make each level distinguishable for this task. Different wording and valid approaches should qualify on their meaning. Do not reward length, a preferred writing style, or reference-text overlap unless those are explicit requirements.

Checks, rubric definitions, metric selectors, and future sequence steps stay outside the target invocation. Put facts the target genuinely needs in explicit task input.

## Read fractional scores without rounding

Jev-backed dimensions retain a fractional score from 0 to 4 in `providerScore.score`. Their ordinal `level` remains null. Scores are not rounded into levels or silently converted into pass/fail decisions.

Set `RUN_ID` to a run returned by `firedrill runs BATCH_ID --json`, replacing BATCH\_ID with your saved batch ID:

```sh theme={null}
firedrill report "$RUN_ID" --json
firedrill assessments list "$RUN_ID" --json
```

The portal report shows the AI score separately from execution, exact constraints, and the optional CI gate. For exact fractional values in CLI output, inspect JSON rather than assuming an ordinal-level summary represents them.

Provider-reported probabilities, confidence, and modal levels are retained metadata. They are not calibrated probabilities that your application is correct. Jev does not supply selected evidence citations or a freeform rationale; do not invent either when explaining a result. Inspect the authored anchors and supplied evidence set.

## Keep exact constraints independent

Use exact checks for schemas, arithmetic, required state transitions, explicit permissions, or a literal protocol value. A check selects an evidence JSON Pointer, comparator, and expected value. Natural-language paraphrases generally should not use exact sentence matching.

The shortage check in [your first simulation](/guides/run-first-drill#5-add-only-the-evaluation-you-need) is an exact constraint. A high semantic score does not erase that check's violation.

An optional gate can require completed execution and satisfied exact checks:

```json theme={null}
"gate": {
  "requireCompleted": true,
  "requireExactChecks": true
}
```

This is a top-level recipe field. Gates are separate policies, not scores. An ordinal `minimumLevels` threshold cannot be satisfied by a fractional-only Jev score; it remains inconclusive. Do not round a score to make a required gate allow. [CI](/guides/ci) documents exit behavior.

## Preserve missing evidence and evaluator errors

Insufficient evidence is `not_assessable`, not zero, full credit, or target failure. Authored non-applicability is separate from missing evidence. Assessment states such as pending, partial, unavailable, or error describe delivery of evaluation, not task quality.

The text evaluator does not assess pixels, audio, or video. A dimension requiring those media stays unassessed; artifact metadata is not the media content. Declare `evidenceKinds` explicitly when a dimension requires nontext evidence.

A fluent output cannot prove an external effect. Use retained Tool state/activity or another attributed observation for claims about actions.

## Reassess without rerunning

Set `RUN_ID` to the retained run you intend to reassess:

```sh theme={null}
firedrill assessments reassess "$RUN_ID" --idempotency-key delivery-review-v1 --json
```

This explicitly requests assessment on frozen evidence, preserving the original assessment. To change the rubric, provide `--rubric rubric.json` with a complete reviewed rubric object and a new deliberate key. Read access alone does not authorize creating assessments.

Reuse the same input and key after uncertainty. Reassessment is not another target execution, and opening a report does not request evaluation.

## Compare repeated observations

Use [repetitions](/guides/simulations) and inspect [batch summaries](/guides/results). Fractional scores have their own count, mean, median, and range within matching rubric/evaluator cohorts; they are not pooled into ordinal bins. Keep missing and unassessable counts visible.

An observed distribution describes those trials. It does not establish statistical significance, universal model ranking, or guaranteed future behavior.

## Prioritize scenario candidates

Your coding agent first drafts candidate scenarios from the repository's actual interfaces and goals. Jev can rank those candidates against explicit criteria; it does not generate, rewrite, save, or execute scenarios.

Save this array as `candidates.json`:

```json theme={null}
[
  {
    "id": "available",
    "name": "Enough stock",
    "input": { "requested": 2, "available": 5 }
  },
  {
    "id": "shortage",
    "name": "Stock shortage",
    "input": { "requested": 6, "available": 5 }
  }
]
```

Save the selection criteria as `criteria.json`:

```json theme={null}
[
  {
    "id": "boundary-coverage",
    "name": "Stock decision boundary coverage",
    "instructions": "Prioritize situations that distinguish approval from review. Do not assume the application has failed.",
    "anchors": [
      { "level": 0, "description": "Unrelated to the stock decision" },
      { "level": 1, "description": "Mentions stock without exercising a decision" },
      { "level": 2, "description": "Exercises an ordinary stock decision" },
      { "level": 3, "description": "Exercises a relevant stock boundary" },
      { "level": 4, "description": "Exercises a distinct high-value boundary for approval versus review" }
    ]
  }
]
```

```sh theme={null}
firedrill scenarios recommend candidates.json --criteria criteria.json \
  --operation-key stock-coverage-v1 --json
firedrill scenarios recommendation --operation-key stock-coverage-v1 --json
```

The first response may be pending. Reads never resend evaluation requests. Keep the same files and operation key after a lost response; changing intent under that key conflicts.

Only candidates with every requested criterion assessed receive an aggregate ranking index: the equal-weight mean of their fractional criterion scores. Ties share rank. Unassessable candidates remain visible and unranked. Ranking is selection guidance, not proof of coverage or an execution outcome.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.