> ## Documentation Index > Fetch the complete documentation index at: https://docs.firedrill.run/llms.txt > Use this file to discover all available pages before exploring further. # Compare retained simulation outcomes and customer measurements > Read one through eight unique batches in this authorized project. Compare exact build and case selections before interpreting outcomes, latency, recorded model usage and reported or explicitly estimated costs. Includes retained failed/cancelled/retry attempt measurements; missing values stay unknown. Reads at most 50,000 interactions and 128 groups with explicit coverage/truncation. Does not run agents, select a winner or change native verdicts. ## OpenAPI ````yaml /api-reference/openapi.json get /v1/projects/{projectId}/simulation-comparisons openapi: 3.1.0 info: title: Firedrill Control API version: 1.0.0 description: >- The control plane for drills, drill runs, and the worlds they run in. Errors always carry the canonical envelope; unsafe operations require an Idempotency-Key; long work returns an operation resource. servers: - url: https://api.firedrill.run security: - controlCredential: [] paths: /v1/projects/{projectId}/simulation-comparisons: get: summary: Compare retained simulation outcomes and customer measurements description: >- Read one through eight unique batches in this authorized project. Compare exact build and case selections before interpreting outcomes, latency, recorded model usage and reported or explicitly estimated costs. Includes retained failed/cancelled/retry attempt measurements; missing values stay unknown. Reads at most 50,000 interactions and 128 groups with explicit coverage/truncation. Does not run agents, select a winner or change native verdicts. operationId: simulations.compare parameters: - name: projectId in: path required: true schema: type: string pattern: ^prj_[0-9a-z]{12,32}$ - name: batchIds in: query required: true schema: type: string maxLength: 328 description: >- One through eight unique simulation batch IDs separated by commas, without whitespace. responses: '200': description: Factual simulation comparison content: application/json: schema: $ref: '#/components/schemas/SimulationComparison' default: description: Canonical error envelope content: application/json: schema: $ref: '#/components/schemas/ErrorEnvelope' components: schemas: SimulationComparison: type: object properties: schemaVersion: type: number enum: - 1 projectId: type: string pattern: ^prj_[0-9a-z]{12,32}$ observedAtMs: type: integer minimum: -9007199254740991 maximum: 9007199254740991 comparability: type: object properties: sameBuild: type: boolean sameCases: type: boolean caseSelectionDigest: type: string pattern: ^sha256:[0-9a-f]{64}$ incompleteBatchIds: maxItems: 8 type: array items: type: string pattern: ^cisuite_[0-9a-z]{12,32}$ required: - sameBuild - sameCases - caseSelectionDigest - incompleteBatchIds additionalProperties: false batches: minItems: 1 maxItems: 8 type: array items: type: object properties: batchId: type: string pattern: ^cisuite_[0-9a-z]{12,32}$ environmentId: type: string pattern: ^env_[0-9a-z]{12,32}$ sourceEnvironmentId: type: string pattern: ^env_[0-9a-z]{12,32}$ planId: type: string pattern: ^simplan_[0-9a-z]{12,32}$ planDigest: type: string pattern: ^sha256:[0-9a-f]{64}$ buildHash: type: string pattern: ^sha256:[0-9a-f]{64}$ state: type: string enum: - preparing - running - cancelling - completed - failed - cancelled counts: type: object properties: logicalCases: type: integer minimum: -9007199254740991 maximum: 9007199254740991 attemptCases: type: integer minimum: -9007199254740991 maximum: 9007199254740991 pending: type: integer minimum: -9007199254740991 maximum: 9007199254740991 active: type: integer minimum: -9007199254740991 maximum: 9007199254740991 passed: type: integer minimum: -9007199254740991 maximum: 9007199254740991 failed: type: integer minimum: -9007199254740991 maximum: 9007199254740991 inconclusive: type: integer minimum: -9007199254740991 maximum: 9007199254740991 error: type: integer minimum: -9007199254740991 maximum: 9007199254740991 cancelled: type: integer minimum: -9007199254740991 maximum: 9007199254740991 required: - logicalCases - attemptCases - pending - active - passed - failed - inconclusive - error - cancelled additionalProperties: false metrics: type: object properties: retainedRunResults: type: integer minimum: -9007199254740991 maximum: 9007199254740991 attemptsWithoutRetainedRunResult: type: integer minimum: -9007199254740991 maximum: 9007199254740991 totalInteractions: type: integer minimum: -9007199254740991 maximum: 9007199254740991 processedInteractions: type: integer minimum: -9007199254740991 maximum: 50000 withMetrics: type: integer minimum: -9007199254740991 maximum: 9007199254740991 missingMetrics: type: integer minimum: -9007199254740991 maximum: 9007199254740991 invalidMetrics: type: integer minimum: -9007199254740991 maximum: 9007199254740991 truncated: type: boolean wallTime: type: object properties: scope: type: string enum: - customer-interaction samples: type: integer minimum: -9007199254740991 maximum: 9007199254740991 unknownInteractions: type: integer minimum: -9007199254740991 maximum: 9007199254740991 p50Ms: nullable: true type: number minimum: 0 p95Ms: nullable: true type: number minimum: 0 minMs: nullable: true type: number minimum: 0 maxMs: nullable: true type: number minimum: 0 required: - scope - samples - unknownInteractions - p50Ms - p95Ms - minMs - maxMs additionalProperties: false recordedModelCalls: type: integer minimum: -9007199254740991 maximum: 9007199254740991 usage: type: object properties: inputTokens: type: object properties: sum: nullable: true type: integer minimum: -9007199254740991 maximum: 9007199254740991 knownCalls: type: integer minimum: -9007199254740991 maximum: 9007199254740991 unknownCalls: type: integer minimum: -9007199254740991 maximum: 9007199254740991 overflow: type: boolean required: - sum - knownCalls - unknownCalls - overflow additionalProperties: false outputTokens: type: object properties: sum: nullable: true type: integer minimum: -9007199254740991 maximum: 9007199254740991 knownCalls: type: integer minimum: -9007199254740991 maximum: 9007199254740991 unknownCalls: type: integer minimum: -9007199254740991 maximum: 9007199254740991 overflow: type: boolean required: - sum - knownCalls - unknownCalls - overflow additionalProperties: false cacheReadInputTokens: type: object properties: sum: nullable: true type: integer minimum: -9007199254740991 maximum: 9007199254740991 knownCalls: type: integer minimum: -9007199254740991 maximum: 9007199254740991 unknownCalls: type: integer minimum: -9007199254740991 maximum: 9007199254740991 overflow: type: boolean required: - sum - knownCalls - unknownCalls - overflow additionalProperties: false cacheCreationInputTokens: type: object properties: sum: nullable: true type: integer minimum: -9007199254740991 maximum: 9007199254740991 knownCalls: type: integer minimum: -9007199254740991 maximum: 9007199254740991 unknownCalls: type: integer minimum: -9007199254740991 maximum: 9007199254740991 overflow: type: boolean required: - sum - knownCalls - unknownCalls - overflow additionalProperties: false required: - inputTokens - outputTokens - cacheReadInputTokens - cacheCreationInputTokens additionalProperties: false costs: maxItems: 128 type: array items: type: object properties: currency: type: string pattern: ^[A-Z]{3}$ basis: type: string enum: - reported - estimated pricingDescription: type: string minLength: 1 maxLength: 512 amount: nullable: true type: number minimum: 0 knownCalls: type: integer minimum: -9007199254740991 maximum: 9007199254740991 overflow: type: boolean required: - currency - basis - amount - knownCalls - overflow additionalProperties: false unknownCostCalls: type: integer minimum: -9007199254740991 maximum: 9007199254740991 benchmark: type: object properties: status: type: string enum: - consistent - mixed - missing value: type: object properties: campaignId: type: string pattern: ^[A-Za-z0-9][A-Za-z0-9._:-]{0,159}$ variantId: type: string pattern: ^[A-Za-z0-9][A-Za-z0-9._:-]{0,159}$ label: type: string minLength: 1 maxLength: 160 pattern: ^[^\u0000-\u001f\u007f]+$ configurationDigest: type: string pattern: ^sha256:[a-f0-9]{64}$ required: - campaignId - variantId - label additionalProperties: false required: - status additionalProperties: false evaluations: maxItems: 128 type: array items: type: object properties: name: type: string minLength: 1 maxLength: 160 pattern: ^[^\u0000-\u001f\u007f]+$ version: type: string minLength: 1 maxLength: 160 pattern: ^[^\u0000-\u001f\u007f]+$ provenance: type: string enum: - customer-evaluator samples: type: integer minimum: 0 exclusiveMinimum: true maximum: 9007199254740991 mean: type: number min: type: number max: type: number minimum: type: number maximum: type: number required: - name - version - provenance - samples - mean - min - max additionalProperties: false models: maxItems: 128 type: array items: type: object properties: provider: nullable: true type: string minLength: 1 maxLength: 160 pattern: ^[^\u0000-\u001f\u007f]+$ requestedModel: nullable: true type: string minLength: 1 maxLength: 160 pattern: ^[^\u0000-\u001f\u007f]+$ responseModel: nullable: true type: string minLength: 1 maxLength: 160 pattern: ^[^\u0000-\u001f\u007f]+$ calls: type: integer minimum: 0 exclusiveMinimum: true maximum: 9007199254740991 required: - provider - requestedModel - responseModel - calls additionalProperties: false groupTruncated: type: boolean required: - retainedRunResults - attemptsWithoutRetainedRunResult - totalInteractions - processedInteractions - withMetrics - missingMetrics - invalidMetrics - truncated - wallTime - recordedModelCalls - usage - costs - unknownCostCalls - benchmark - evaluations - models - groupTruncated additionalProperties: false required: - batchId - environmentId - planId - planDigest - buildHash - state - counts - metrics additionalProperties: false required: - schemaVersion - projectId - observedAtMs - comparability - batches additionalProperties: false ErrorEnvelope: type: object properties: schemaVersion: type: number enum: - 1 code: type: string pattern: ^[a-z][a-z0-9-]*\.[A-Z][A-Z0-9_]*$ source: type: string enum: - framework - world - tool - target - assertion - reporter - control message: type: string minLength: 1 maxLength: 4000 retryable: type: boolean correlationId: type: string pattern: ^corr_[A-Za-z0-9][A-Za-z0-9_-]{5,95}$ issues: default: [] type: array items: type: object properties: code: type: string minLength: 1 maxLength: 128 message: type: string minLength: 1 maxLength: 2000 path: type: array items: anyOf: - type: string - type: integer minimum: 0 maximum: 9007199254740991 suggestion: type: string minLength: 1 maxLength: 4000 required: - code - message additionalProperties: {} details: type: object additionalProperties: $ref: '#/components/schemas/ErrorEnvelope___schema0' evidence: type: object properties: runId: type: string pattern: ^run_[A-Za-z0-9][A-Za-z0-9_-]{5,95}$ sequence: type: integer minimum: 0 exclusiveMinimum: true maximum: 9007199254740991 additionalProperties: {} required: - schemaVersion - code - source - message - retryable - issues additionalProperties: {} ErrorEnvelope___schema0: description: Any JSON-compatible value securitySchemes: controlCredential: type: http scheme: bearer description: Opaque control credential ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.