Skip to main content
Add evaluation only when it helps answer your question. A simulation can remain observation-only. For assessed runs, use semantic rubrics for meaning and exact checks for obligations that must hold precisely. Your coding agent authors the task and criteria. Firedrill’s Jev evaluator assesses retained text evidence; it does not execute your target or its Tools. The target’s model and credentials remain in your own runner.

​
Write meaningful anchors

Add a rubric to a scenario used with your actual language-producing application. This complete scenario asks for a factual delivery update without prescribing one sentence:
Place the object in your recipe’s scenarios array, using your application’s real command, HTTP endpoint, or native callback. The deterministic stock adapter in the first-simulation guide does not implement this language task. The five anchors define positions 0–4. Make each level distinguishable for this task. Different wording and valid approaches should qualify on their meaning. Do not reward length, a preferred writing style, or reference-text overlap unless those are explicit requirements. Checks, rubric definitions, metric selectors, and future sequence steps stay outside the target invocation. Put facts the target genuinely needs in explicit task input.

​
Read fractional scores without rounding

Jev-backed dimensions retain a fractional score from 0 to 4 in providerScore.score. Their ordinal level remains null. Scores are not rounded into levels or silently converted into pass/fail decisions. Set RUN_ID to a run returned by firedrill runs BATCH_ID --json, replacing BATCH_ID with your saved batch ID:
The portal report shows the AI score separately from execution, exact constraints, and the optional CI gate. For exact fractional values in CLI output, inspect JSON rather than assuming an ordinal-level summary represents them. Provider-reported probabilities, confidence, and modal levels are retained metadata. They are not calibrated probabilities that your application is correct. Jev does not supply selected evidence citations or a freeform rationale; do not invent either when explaining a result. Inspect the authored anchors and supplied evidence set.

​
Keep exact constraints independent

Use exact checks for schemas, arithmetic, required state transitions, explicit permissions, or a literal protocol value. A check selects an evidence JSON Pointer, comparator, and expected value. Natural-language paraphrases generally should not use exact sentence matching. The shortage check in your first simulation is an exact constraint. A high semantic score does not erase that check’s violation. An optional gate can require completed execution and satisfied exact checks:
This is a top-level recipe field. Gates are separate policies, not scores. An ordinal minimumLevels threshold cannot be satisfied by a fractional-only Jev score; it remains inconclusive. Do not round a score to make a required gate allow. CI documents exit behavior.

​
Preserve missing evidence and evaluator errors

Insufficient evidence is not_assessable, not zero, full credit, or target failure. Authored non-applicability is separate from missing evidence. Assessment states such as pending, partial, unavailable, or error describe delivery of evaluation, not task quality. The text evaluator does not assess pixels, audio, or video. A dimension requiring those media stays unassessed; artifact metadata is not the media content. Declare evidenceKinds explicitly when a dimension requires nontext evidence. A fluent output cannot prove an external effect. Use retained Tool state/activity or another attributed observation for claims about actions.

​
Reassess without rerunning

Set RUN_ID to the retained run you intend to reassess:
This explicitly requests assessment on frozen evidence, preserving the original assessment. To change the rubric, provide --rubric rubric.json with a complete reviewed rubric object and a new deliberate key. Read access alone does not authorize creating assessments. Reuse the same input and key after uncertainty. Reassessment is not another target execution, and opening a report does not request evaluation.

​
Compare repeated observations

Use repetitions and inspect batch summaries. Fractional scores have their own count, mean, median, and range within matching rubric/evaluator cohorts; they are not pooled into ordinal bins. Keep missing and unassessable counts visible. An observed distribution describes those trials. It does not establish statistical significance, universal model ranking, or guaranteed future behavior.

​
Prioritize scenario candidates

Your coding agent first drafts candidate scenarios from the repository’s actual interfaces and goals. Jev can rank those candidates against explicit criteria; it does not generate, rewrite, save, or execute scenarios. Save this array as candidates.json:
Save the selection criteria as criteria.json:
The first response may be pending. Reads never resend evaluation requests. Keep the same files and operation key after a lost response; changing intent under that key conflicts. Only candidates with every requested criterion assessed receive an aggregate ranking index: the equal-weight mean of their fractional criterion scores. Ties share rank. Unassessable candidates remain visible and unranked. Ranking is selection guidance, not proof of coverage or an execution outcome.