Public benchmarks can shortlist a model. Your product eval has to decide

Artificial Analysis has launched Optima, a product for building benchmarks from a team’s own files, agent traces, coding environment, or task descriptions. Its pitch is deliberately practical: compare models on quality, cost per task, and time per task instead of assuming a public leaderboard can select the right system for a particular workflow (official announcement).

That is a useful change in interface. It is not a change in epistemology. A private dataset can still be unrepresentative. A generated rubric can reward the wrong behavior. A model judge can agree with itself and disagree with users. A task can leak its answer, a harness can favor one provider, and a single run can turn sampling luck into a purchase decision.

The immediate lesson for a small AI product team is therefore not “move every benchmark into Optima.” It is: stop buying a model score and start maintaining a decision instrument.

This field note proposes a private product-eval promotion protocol. It uses a narrow decision contract, three task banks, an evidence-custody ledger, grader calibration, wording variants, repeated trials, accepted-task economics, and a signed promotion receipt. Y Build did not run the proposed experiment, and none of the blank result fields below represents observed performance. Optima’s launch and examples are vendor claims; the other projects show useful design boundaries, not proof that this protocol will work unchanged in every product.

What Optima changes—and what it leaves with your team

Optima’s product page says a team can import evaluation files, Hugging Face datasets, agent traces from supported platforms, or a coding environment. It also says Optima can build a benchmark from a use-case description and example inputs and outputs. Separately, the platform supports rubric-based or pairwise grading while reporting performance, cost per task, and time per task (Optima product page).

Those capabilities remove real setup friction. A three-person team should not need to build a distributed evaluation runner before it can compare two model candidates. Importing actual traces is also more promising than starting from generic trivia questions.

But the platform cannot decide what your evidence means. Consider five unresolved choices:

1. Which production population do the imported traces represent?
2. Which rare failures deserve more weight than their frequency suggests?
3. Does the rubric measure a customer outcome or merely a neat answer?
4. Can the chosen grader distinguish an acceptable alternative from an error?
5. Does better average quality compensate for slower recovery, higher variance, or a severe failure?

The launch material includes example benchmark results and claims that users can identify alternatives with large cost or time savings. Treat those as demonstrations of the product surface, not independent model findings. The results depend on the tasks, model versions, harness, grader, and run date. The relevant unit is the complete evaluation configuration, not the model name at the top of a chart.

Narrow benchmarks answer better questions, not every question

Two other current projects make the value and the limit of narrowing easier to see.

MathCode is a terminal agent built around one demanding workflow: translate a natural-language mathematics problem into a Lean 4 theorem and attempt a formal proof. A compiled proof is a much stronger terminal artifact than a fluent explanation. Yet MathCode’s success on formalization would not establish that the same system can handle support triage, contract review, or product research. The narrow verifier strengthens a narrow claim.

LittleLearner narrows a different boundary. The researchers trained a 5-billion-parameter model from scratch on an 88-billion-token corpus filtered to material within a U.S. elementary-school curriculum. They present it as a controlled sandbox for studying what models acquire inside and outside a defined training scope. Their initial experiments found that post-training and in-context learning helped the model use existing knowledge but did not meaningfully lift out-of-scope capability.

That is research evidence for a particular controlled environment, not a purchasing result. It is useful here because the training boundary is legible. Product evaluations should aim for the same kind of claim discipline:

• “Candidate B handled our 24 refund-policy fixtures under the pinned configuration” is bounded.
• “Candidate B is the smartest support model” is not.
• “No severe policy violation appeared in 72 declared trials” is evidence.
• “Candidate B is safe” is not.

A narrow evaluation earns credibility by reducing the size of the conclusion. It should not be used to smuggle a larger conclusion through a smaller test.

Evaluate the product system, not the model label

A customer never experiences a base model in isolation. They experience a system. Public frameworks such as HELM standardize adaptation when comparing models; a private product evaluation has the opposite additional duty of capturing the particular adaptation customers will receive.

system outcome = model snapshot
               + prompts and policies
               + tool definitions and permissions
               + retrieved context and customer state
               + orchestration, retries, and fallbacks
               + user interface and recovery path

Anthropic’s current guide to agent evaluations makes this distinction explicit: an agent harness runs the model, tools, and loop, while an evaluation harness runs tasks, records trials, applies graders, and aggregates results. When a team says it evaluated “Model A,” it often evaluated Model A inside one scaffold, with one tool schema and one budget.

Freeze the full system fingerprint for every candidate:

If any material field changes, the receipt expires. A stable marketing name does not preserve the evaluated system.

Start with one promotion decision

Imagine LedgerLane, a small SaaS product that reads inbound expense receipts and drafts accounting-review packets. The team is considering a new model because it appears faster and cheaper in public comparisons. The production workflow includes image extraction, vendor matching, duplicate detection, policy lookup, anomaly explanation, and a human approval queue.

The team should not begin with “Which model is best?” It should write a decision:

Can candidate system B replace the current system for English-language receipts from the three supported markets, without increasing severe policy misses, while reducing accepted-packet cost or review time?

That sentence defines a population, an incumbent, a candidate, a risk boundary, and the kinds of benefit that count. It also exposes exclusions. The test does not establish multilingual quality, tax advice, fraud detection, autonomous payment approval, or behavior on document formats outside the supported markets.

Write four possible outcomes before generating tasks:

• hold: evidence is invalid, incomplete, or below a blocking threshold;
• shadow: run the candidate without customer-visible output to gather production-shaped evidence;
• limited_go: expose a named low-risk segment with automatic fallback;
• promote: make the candidate the default only for the declared scope.

Without predetermined states, teams are tempted to turn any interesting score into “ship.”

Build three task banks with different jobs

A single pile of examples creates a single average that conceals why the test exists. Maintain three banks.

1. Core acceptance bank

Use 20–50 ordinary tasks drawn from real manual checks, support issues, and product requirements. Anthropic recommends this range as a practical starting point for early agent evals, while noting that mature products need larger sets to detect smaller effects. For LedgerLane, core cases include clear receipts, known vendors, common currencies, expected policy fields, and ordinary duplicate states.

The core bank answers: can the candidate reliably do the work customers already depend on? Its pass rate should be high enough to serve as a regression gate.

2. Edge and harm bank

Collect rare or costly conditions: cropped totals, conflicting dates, duplicated uploads across workspaces, prompt-like text inside a receipt, unsupported tax claims, missing currency, ambiguous vendor aliases, and an attempted write outside the approval boundary.

Do not let frequency erase severity. One autonomous payment, one cross-tenant disclosure, or one invented compliance claim can block promotion even if 99 routine cases pass.

3. Capability frontier bank

Add tasks the incumbent often fails but the product would benefit from solving: multi-page bundles, handwritten notes, mixed-language fields, or a constrained explanation that cites the exact policy clause. A low initial pass rate is acceptable. The bank asks whether a candidate creates a useful new frontier, not whether it preserves existing behavior.

Keep the banks separate in reporting. A model can improve frontier capability while regressing the product’s core contract. Averaging them together converts an important trade-off into a comforting number.

Separate learning evidence from promotion evidence

The three banks describe why cases exist. Each bank also needs three access partitions that describe how cases may be used.

Assign cases to partitions before tuning. Store content hashes and split IDs. Do not place near-duplicates, paraphrases, or the same customer incident across partitions. Keep a salted similarity audit or manual family identifier so one underlying example cannot become both the lesson and the exam.

The promotion holdout should be access-controlled. Builders can know its declared categories, counts, severity mix, and grading method, but not the case contents or expected outcomes. The custodian records every access. If a builder sees a holdout case, the case moves out of the holdout; it does not remain “blind” because nobody remembers changing a spreadsheet.

After the holdout is opened, the receipt has one of two outcomes: accept the frozen comparison or tune the system and require fresh promotion evidence. Reusing the same holdout after prompt, rubric, tool, threshold, or harness changes turns a release gate into a development set. A small team may rotate retired holdout cases into regression coverage, but it should replenish the blind partition from newly consented failures, newly written synthetic cases, or a separately maintained reserve.

This is the protocol’s main difference from a generic grader guide. It assumes the team already established that graders are sufficiently independent and calibrated. It governs whether the private benchmark itself remains valid enough to authorize a promotion decision.

Keep private evidence useful without turning it into a black box

Private tasks reduce the chance that a model memorized a public answer and make the suite more relevant to the product. They also create custody, privacy, and audit problems.

OpenAI’s recent third-party evaluation playbook recommends private or newly constructed tasks where possible for contamination resistance, but also calls out broken problems, harness effects, shortcuts, refusals, and sample review. Privacy solves only one validity threat. A contemporary independent summary of Optima reaches the same practical boundary: use-case relevance improves, but methodology problems such as data quality and evaluation bias do not disappear (The Decoder).

For every fixture, keep a record with:

Use synthetic replacements where customer content is unnecessary. If authentic examples are needed, minimize fields, restrict access, set retention, and separate the reusable fixture from identifying raw material. Do not upload production traces to a hosted evaluation service until its data handling, retention, training use, access control, deletion, and regional terms match the team’s obligations.

The public report can disclose task categories, partition counts, hashes, grader types, repeat policy, access exceptions, and limitations without publishing private records or answer keys. “Private” must not mean “unreviewable.”

Make every grader answerable to another form of evidence

Different claims need different graders.

Deterministic checks should verify fields, schemas, database state, permission boundaries, numeric reconciliation, citations, and other facts that software can inspect directly. For LedgerLane, the claimed total must equal the captured line items; the vendor must exist; a duplicate must not create a second draft; no payment state may change.

Model graders can help with explanation quality, policy relevance, and whether a draft includes an important caveat. They need a versioned rubric and a human calibration sample. Do not use the candidate itself as the only grader of its own output; this protocol assumes that grader-independence work is complete before the promotion holdout opens.

Human review should own ambiguous, high-severity, and taste-dependent cases. A domain reviewer must be able to choose accept, reject, or abstain, record the reason, and appeal a rubric that rewards the wrong thing.

Product outcomes belong after controlled offline validity. Shadow-mode reconciliation, review time, correction rate, abandonment, and user feedback can reveal whether the offline score represents value.

OpenAI’s contextual-evaluation guidance recommends a golden set built with domain experts, realistic conditions, rare costly edges, regular auditing of model graders, and continuous incorporation of production signals (Specify → Measure → Improve). NIST similarly says test sets, metrics, and tools should be documented and performance demonstrated under conditions similar to deployment (AI RMF Measure).

The practical rule is simple: no grader is sovereign. Deterministic checks constrain model judges, people calibrate subjective rubrics, and production outcomes test whether the offline construct mattered.

Vary wording and repeat trials before trusting a rank

One task phrasing is not a task distribution. The new BenchDrift study generated meaning-preserving changes along linguistic, referential, pragmatic, and structural axes. Across eight models and three benchmarks, answers flipped in both directions; the authors report that stronger models did not become immune to wording sensitivity (paper, code and data).

Do not transfer those exact drift rates to product prompts. Use the result to challenge a fragile design assumption.

For each high-value LedgerLane intent, create four variants:

1. ordinary production wording;
2. shorter wording with the same requirement;
3. reordered context and constraint placement;
4. a realistic referential variant using pronouns, prior-turn state, or UI labels.

Have a domain reviewer confirm that each variant preserves the expected behavior. If the wording changes the requirement, it is a new task, not a robustness variant.

Then run at least three trials per variant for non-deterministic systems. Rotate candidate order, and reset every trial from the same clean seeded snapshot rather than carrying state from the prior run. Separate first-pass success from recovery-assisted success. Report the distribution and the worst meaningful slice, not only an average.

The proposed minimum is intentionally modest: 30 base tasks × 4 variants × 3 trials × 2 candidates = 720 trials. A team with a smaller budget can start with fewer base tasks, but it should preserve categories, variants, repeats, and severe cases before adding breadth.

Add economics only after validity gates

Optima’s decision to show cost per task and time per task beside quality is directionally right. Only compare economics after the holdout remains blind, graders pass calibration, no blocking failure occurs, and the candidate clears the core bank. Then compute the cost of an accepted outcome.

accepted-task cost =
  (model + tool + retrieval + grader + retry + human-review cost)
  / independently accepted tasks

Keep first-pass and recovery-assisted acceptance separate; report p50/p95 time, human-review minutes, cost by bank, and the actual model used after fallback.

A cheap model that doubles review time can be more expensive. A fast model that produces a rare unacceptable state can be unshippable. Predeclare the smallest cost, time, or frontier improvement worth operational change; a positive but trivial point estimate is not a benefit.

Do not monetize a severe safety or privacy failure merely to average it into the denominator. Blocking constraints remain constraints.

Use one versioned evaluation contract

Store the experiment definition before the first candidate run. A reusable minimum looks like this:

eval_id: ledgerlane-receipt-review-2026-08
decision: replace-incumbent-for-supported-english-receipts
population:
  markets: [market_a, market_b, market_c]
  languages: [en]
excluded_claims:
  - tax_advice
  - fraud_detection
  - payment_approval

systems:
  incumbent: {model: null, prompt_hash: null, harness_sha: null}
  candidate: {model: null, prompt_hash: null, harness_sha: null}

evidence_custody:
  dataset_split_ids: {development: null, calibration: null, promotion_holdout: null}
  dataset_hashes: {development: null, calibration: null, promotion_holdout: null}
  holdout_custodian: null
  access_log: null
  contamination_check: null
  post_open_tuning_requires_fresh_receipt: true

task_banks:
  core: {base_tasks: 20, variants_each: 4, trials_each: 3}
  edge: {base_tasks: 8, variants_each: 4, trials_each: 3}
  frontier: {base_tasks: 2, variants_each: 4, trials_each: 3}

graders:
  deterministic: [schema, total_reconciliation, duplicate_state, no_payment_write]
  model_rubric: {model: null, prompt_hash: null, calibrated_cases: 12}
  human_review: {owner: null, mandatory_cases: [all_severe, all_abstain]}
  blind_adjudication: {enabled: true, candidate_labels_hidden: true}

blocking_gates:
  severe_policy_misses: 0
  cross_tenant_disclosures: 0
  unauthorized_writes: 0
  core_regressions_vs_incumbent: 0

benefit_gate:
  require_one_of: [lower_accepted_task_cost, lower_review_time, frontier_gain]
  minimum_meaningful_delta: null
  comparison_method: paired_by_fixture_with_interval
  uncertainty_rule: hold_if_interval_crosses_predeclared_delta

signoff:
  decision_owner: null
  independent_reviewer: null

result:
  decision: null
  evidence_bundle: null
  expires_on: null
  rerun_triggers: []

The zeros are zero-observed release gates, not statistical proof that an unseen failure cannot occur in the population. Record the trial denominator. If a benefit is smaller than the predeclared meaningful delta, or its paired interval crosses that delta, choose hold or shadow; do not promote on a noisy point estimate. The named decision owner signs the product decision, and an independent reviewer signs the evidence-validity judgment.

Treat the evaluation itself as a product with failure modes

Task-generation monoculture. One model drafts the tasks, expected answers, and rubric. Its preferences become invisible ground truth. Require human review and seed the bank with real requirements and failures.

Trace popularity bias. Imported traces overrepresent frequent successful users and recent instrumentation. Deliberately sample abandoned, corrected, escalated, slow, and low-frequency high-impact cases.

Answer leakage. File names, comments, hidden state, repository history, web access, or a near-duplicate development case exposes the target. Inspect the environment, audit case families across splits, and test whether a candidate can retrieve task-specific details before solving.

Rubric gaming. The grader rewards mentioning policy vocabulary without taking the correct action. Pair surface-quality rubrics with terminal-state checks.

Pairwise inconsistency. A judge prefers A to B, B to C, and C to A, or changes preference when order flips. Randomize order, measure disagreement, and send unstable cases to people.

Regression masking. Large frontier gains outweigh small core losses in a composite score. Keep banks and blocking constraints separate.

Privacy theater. The suite is called private, but raw customer content is copied into multiple vendors, logs, spreadsheets, and reviewer inboxes. Maintain a data-flow and deletion record.

Holdout reuse. A failed run leads to tuning and another attempt on the same supposedly blind cases. Retire exposed cases and require fresh promotion evidence.

Snapshot evaporation. A hosted alias, evaluator, or routing policy changes after approval. Expire the receipt on any material fingerprint change.

Benchmark ownership vacuum. Nobody removes bad tasks, investigates disagreement, or adds production failures. Assign an owner, review cadence, and change log.

Know when this protocol is too much—or not enough

Use the full protocol when changing a model, provider, reasoning budget, agent harness, tool permission, retrieval system, or grader could alter customer-visible decisions or mutable state. It is especially useful when a public benchmark cannot represent proprietary workflow details.

For an internal drafting aid with mandatory human review and no sensitive data, a smaller suite may be appropriate: preserve the decision contract, task banks, model fingerprint, grader calibration, repeated trials, and result receipt, but reduce the number of variants and reviews.

Do not use this protocol as certification for medical, legal, financial, employment, education, or other regulated decisions. It does not prove fairness, privacy compliance, security, or general safety. Domain-specific validation, affected-user research, security testing, legal review, and ongoing monitoring may be required.

It is also insufficient for a new product category with no stable definition of success. In that case, conduct user research and clarify the workflow before optimizing a benchmark. A precise score for the wrong product promise is not progress.

A 48-hour Build Lab plan

Hours 0–4: write the decision, population, exclusions, promotion states, and blocking constraints. Name the decision owner and evaluation owner.

Hours 4–12: collect 20–30 safe base cases across core, edge, and frontier banks. Assign development, calibration, and holdout partitions; record hashes, provenance, consent, redaction, expected behavior, access, and expiry triggers.

Hours 12–18: implement deterministic checks first. Draft subjective rubrics only for claims that cannot be checked from state or structure. Calibrate on at least 12 mixed cases with a domain reviewer.

Hours 18–24: generate wording variants, review semantic equivalence, freeze the system fingerprints and environment, and calculate the trial budget.

Hours 24–36: freeze configuration, open the blind holdout, and run both candidates with rotated order and clean environment resets. Quarantine broken fixtures instead of silently editing them mid-run.

Hours 36–42: inspect every severe failure, grader disagreement, abstention, and large candidate delta. Compute accepted-task cost and review time by bank.

Hours 42–48: sign hold, shadow, limited_go, or promote for the declared scope. Attach the evidence bundle, limitations, expiry date, and rerun triggers. If the evidence is invalid, the correct result is hold, not a repaired chart.

The durable asset is the decision instrument

Optima makes custom benchmarking more accessible. MathCode shows the value of a verifier tied to a narrow workflow. LittleLearner shows how a controlled scope can make capability boundaries more interpretable. BenchDrift warns that one wording can produce a brittle score.

Together they support a restrained conclusion: the best evaluation is not the largest leaderboard or the most private dataset. It is the smallest maintained instrument that represents the product decision, exposes its blind spots, and can be rerun when the system changes.

Public benchmarks still matter. They help discover candidates, reveal broad capability, and supply shared reference points. Let them shortlist. Let your versioned product evaluation decide—and make the receipt expire.

References

1. Artificial Analysis — Announcing Optima: create a custom benchmark for your use case
2. Artificial Analysis — Optima product page
3. The Decoder — Optima lets users test models against their own data
4. Math-AI — MathCode repository
5. Li et al. — LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure
6. Thakur et al. — The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance
7. IBM — BenchDrift code and data
8. Anthropic — Demystifying evals for AI agents
9. OpenAI — How evals drive the next chapter in AI for businesses
10. OpenAI — A shared playbook for trustworthy third-party evaluations
11. NIST — AI Risk Management Framework Core
12. Stanford CRFM — Holistic Evaluation of Language Models