Your research agent finished the report. Audit the decisions, not the document

An AI research agent opens hundreds of papers, writes working analysis code, launches experiments, generates charts, asks other agents for reviews, and returns a polished report before the deadline. That looks like successful delegation. It may only prove that the workflow ran.

CRUX recently gave frontier agents the central questions from two unpublished NeurIPS 2026 submissions. Each main run received six days, up to $3,000 in API credits, GPU access, a virtual machine, the open web, and multiple review channels. The agents completed the engineering and produced papers. The original researchers rated one paper Reject and the other Strong Reject. A robustness run using a different model and native scaffold reproduced similar failure patterns. The CRUX evaluation attributes the gap to judgment: weak evidence choices, premature commitment, poor backtracking, ineffective response to criticism, resource blindness, and instruction drift.

This is early evidence, not a universal verdict. The study covers two research questions and five runs. Original authors reviewed the output non-blindly. The tasks targeted top-conference AI research, not ordinary product discovery. CRUX explicitly documents these limits in its paper and methodology.

Still, the result creates a practical decision for small teams using agents for market research, user synthesis, competitive analysis, incident investigation, or product experiments. Do not promote a research agent because it can complete a research-shaped pipeline. Promote it only after you can inspect whether it made defensible decisions when the answer was ambiguous.

This field note proposes a research-agent judgment audit built around a decision-fork ledger, a disconfirmation gate, and a review-response test. Y Build did not run the proposed audit. Every threshold and result field below is a starting point for a team to calibrate, not an observed benchmark result.

What the CRUX evaluation actually tested

The study introduced a shadow evaluation. An agent received a real research question from a strong paper that had not yet been published, but not the paper’s findings. The agent therefore could not retrieve the answer from the web or reproduce a known manuscript. The original authors later reviewed the generated paper because they had spent months exploring the same question.

The two questions concerned controllability of language-model personas and distribution-shift detection for tabular foundation models. The main runs used Claude Opus 4.8 with extra-high reasoning through OpenClaw. The team then repeated one problem with GPT-5.6 Sol and Codex to test whether the initial failures were mostly a scaffold artifact. CRUX releases its code, data, and available agent logs, although one unpublished problem necessarily has restricted detail.

The headline result was not that the agents did nothing useful. Reviewers credited literature coverage, large amounts of functioning experimentation, reasonable initial hypotheses, and a few potentially interesting findings. The failure was that the final claims did not earn the strength, novelty, or significance assigned to them.

That distinction matters. A product team may receive a beautifully structured competitive brief whose source selection cannot support the recommendation. A user-research agent may cluster interviews correctly but overstate what six participants establish. An incident agent may reproduce a symptom yet settle on the first plausible cause. In all three cases the document can be complete while the decision is unsafe.

Separate research execution from research judgment

Existing research-agent benchmarks deliberately measure different constructs.

CORE-Bench asks agents to reproduce computational results across 270 tasks derived from 90 papers. MLE-bench uses 75 Kaggle competitions with deterministic task grading and recommends repeated runs because agent behavior remains variable. PaperBench decomposes replication of 20 ICML papers into 8,316 gradable subtasks. These are valuable tests of reading, environment setup, implementation, experimentation, and evidence recovery.

RE-Bench moves closer to research engineering: seven novel environments, human comparisons, fixed time budgets, and measurable optimization targets. METR reported that agents could generate and test implementations much faster than people, sometimes beating humans at two hours, while performing worse at longer budgets and often failing to adapt to novel information.

None of these results is invalidated by CRUX. They answer narrower questions:

A serious evaluation needs more than one row. The product mistake is collapsing all four into “the agent did research.”

Why a finished artifact is an especially convincing false positive

Research workflows produce visible activity. Tokens, papers read, code commits, GPU hours, charts, citations, and pages written are easy to count. Judgment appears mainly at branch points: which question to narrow, which confounder to test, whether a negative result is informative, when to abandon a method, and how much confidence the evidence deserves.

The CRUX agents received repeated critical feedback. Their own review processes did not produce an acceptance. Yet the typical response was to add limitations or polish the existing direction instead of redesigning the weak experiment. The criticism existed in the trace, but it did not change the trajectory.

That is a dangerous product pattern because review can become theater. A dashboard can show “three reviewers consulted” while none has power to force a new decision. A report can contain a thoughtful limitations section while its headline retains the unsupported claim. A budget meter can be visible while the system cannot translate remaining money or time into a better next experiment.

OpenAI’s recent field report on eight agent-assisted scientific-computing projects describes a compatible boundary from a different setting: agents accelerated well-scoped engineering, while people still had to specify acceptance targets, judge scientific validity, handle edge cases, and own long-term stewardship. That report is retrospective and contributor-written, not a controlled comparison. Together with CRUX, it supports a restrained operational inference: execution throughput and decision quality should be measured separately.

Use one product question with no lookup-table answer

Build the audit around a real but non-live product decision. Imagine a small team running a trial for Harbor, an AI inbox that summarizes support conversations. Activation is acceptable, but many new teams stop using the product during the second week. The research question is:

Which one product change is most likely to reduce week-two abandonment for newly activated teams, and what evidence would justify a limited experiment?

This is not a web-search question. Existing studies may supply hypotheses, but Harbor’s users, product state, and instrumentation determine the answer. The agent receives a synthetic research pack:

• twelve redacted interview excerpts, including contradictory feedback;
• an event dictionary and a small de-identified funnel table;
• three support-ticket clusters;
• four competitor onboarding flows captured on fixed dates;
• known instrumentation gaps;
• a two-day wall-clock limit and a fictional $200 research budget;
• authority to analyze and draft, but not contact users or change production.

Create the pack from safe historical or synthetic material. Do not add personal data merely to make the task realistic. The correct output is not a predetermined feature. It is a defensible recommendation, calibrated uncertainty, an alternative interpretation, and a next experiment that could disconfirm the recommendation.

The lack of a single correct answer is intentional. You are testing whether the agent keeps uncertainty legible, not whether it guesses a hidden label.

Freeze the brief and the decision rights before the run

Write a one-page contract that the agent cannot silently rewrite. Include:

The brief must distinguish a research question from a delivery request. “Write a retention report” rewards the production of a report. “Identify the best-supported explanation, record viable alternatives, and propose a disconfirming experiment” rewards epistemic work.

Also define who owns the final decision. An agent can recommend continue, reframe, abandon, or collect_more. It cannot convert its recommendation into a product roadmap item. This keeps the audit about judgment support rather than delegated authority.

Capture every meaningful fork in a decision ledger

Logs of tool calls are necessary but too granular. A decision fork is a moment when two plausible research paths existed and the agent chose one. Require the agent to append a structured record before the choice takes effect.

research_decision:
  run_id: harbor-judgment-01
  fork_id: F04
  observed_at: "2026-08-15T03:20:00Z"
  question: "Why does usage fall after day seven?"
  current_hypothesis: "Teams do not trust summaries enough to reuse them"
  alternatives:
    - "Teams cannot find the recurring-summary workflow"
    - "The week-two event is under-instrumented"
    - "Interview sample overrepresents admins"
  evidence_for:
    - artifact: interview-pack-v3
      locator: excerpt-07
      strength: weak
  evidence_against:
    - artifact: funnel-table-v2
      locator: rows-18-31
      strength: medium
  unknowns:
    - "No role-segmented retention data"
  options:
    - id: O1
      action: "analyze navigation evidence"
      expected_information_gain: medium
      cost_minutes: 35
    - id: O2
      action: "draft trust intervention"
      expected_information_gain: low
      cost_minutes: 25
  selected: O1
  selection_reason: "Tests a live alternative before intervention design"
  disconfirming_observation: "Drop-off is unchanged among users who find the workflow"
  revisit_at: "after role-segmented navigation analysis"
  decision_owner: product-lead

Do not ask for a ledger entry after every command. Capture it when the agent retires a hypothesis, changes the population, chooses a proxy, selects an experiment, ignores a review, or makes a recommendation. Six to twelve meaningful forks are more useful than ten thousand lines of terminal history.

The ledger makes a subtle failure inspectable: an agent may list alternatives yet repeatedly choose the option that preserves its first story. That is not diversified exploration.

Add a disconfirmation gate and a review-response test

Before the agent may write the recommendation, require three artifacts.

First, a claim-evidence table:

Second, a disconfirmation run. The agent must spend a predeclared share of time—start with 20%, then calibrate—trying to break its leading explanation. This is not a universal statistical rule. It is a forcing function against premature convergence. Valid work includes re-segmenting the data, seeking negative cases, testing an alternative proxy, or asking whether missing telemetry could produce the same pattern.

Third, a review-response diff. Give a reviewer only the frozen brief, evidence pack, ledger, and draft. The reviewer marks each issue as blocking, material, or optional. The agent must answer with one of four responses:

• changed_method: ran a different analysis or experiment;
• changed_claim: narrowed or removed a claim;
• added_evidence: supplied traceable new support;
• disagreed: retained the decision with a specific reason and evidence.

“Added a caveat” is not a fifth response. It may be part of changed_claim, but it cannot preserve the same unsupported conclusion. CRUX’s strongest product lesson is that detecting criticism and acting on criticism are separate capabilities.

Score judgment independently from output quality

Use two scorecards. The first covers execution: source retrieval, code correctness, reproducibility, citation integrity, artifact completeness, and compliance with the brief. The second covers judgment:

Require a human reviewer to cite ledger fork IDs and artifact locations for every score. Do not let the acting agent grade itself. NIST’s AI RMF Measure guidance recommends documented test methods, uncertainty, deployment-relevant conditions, and assessors beyond front-line developers. It also warns teams to examine whether a proxy measures the intended construct. Here, document quality is the tempting proxy; research judgment is the intended construct.

A starting promotion rule might require no zero in any judgment dimension, at least 11 of 14 points, and no blocking evidence or instruction violation. Those numbers are proposed calibration anchors. Run several human and agent baselines before treating them as meaningful.

Run six failure cases before trusting the happy path

The Harbor pack can produce six controlled tests:

1. Seductive first hypothesis: place the most fluent interview quote early, while stronger contradictory evidence appears later. Check whether order controls the conclusion.
2. Broken proxy: include an event named summary_saved that does not reliably mean a user consumed the summary. Check whether the agent verifies the event definition.
3. Underpowered negative: provide a tiny subgroup with no apparent difference. Check whether the agent treats absence of signal as evidence of no effect.
4. Late material review: after the first draft, point out that admins dominate the interviews. Check whether the agent changes the method or merely adds a limitation.
5. Unused budget: leave time and budget after a weak result. Check whether the agent chooses a high-information next step instead of ending early or generating more prose.
6. Instruction conflict: add an enticing public competitor dataset while the brief allows only fixed-date captures. Check whether the agent asks for scope approval or drifts.

Run each case independently from the same baseline. Repeat enough times to observe variance; MLE-bench’s maintainers recommend at least three seeds for their benchmark because deterministic grading does not make agent behavior deterministic. Your audit is smaller and more qualitative, so avoid pretending three runs establish a population rate. Repetition reveals unstable branches and gives the reviewer concrete traces to compare.

Know the failure modes the audit still cannot eliminate

A good ledger can itself become a polished artifact with little influence on behavior. Check timestamps: was the entry created before the choice, or reconstructed after the result? Check alternatives: are they genuinely plausible, or straw options written to justify the preferred route? Check references: does a citation support the exact claim, or only the surrounding topic?

Other limits remain:

• Reviewer dependence: experts can disagree about novelty, significance, or what evidence is sufficient. NeurIPS 2026’s reviewer criteria separate quality, clarity, significance, and originality precisely because one polished score cannot represent all four.
• Task leakage: a fixed audit pack can become familiar to a model or scaffold. Rotate scenarios and keep some evidence variants private.
• Scaffold dependence: planning tools, memory, budget telemetry, and review routing can materially affect behavior. Record them as part of the system under test.
• Domain limits: a result on retention research does not authorize medical, legal, financial, security, or scientific conclusions. Those require qualified domain review and applicable governance.
• No ground truth: open-ended work often lacks a single correct decision. The audit can expose reasoning and evidence discipline; it cannot convert judgment into certainty.

The opposite category error matters too. A judgment audit is unnecessary for every task. If the job is to reproduce a fixed calculation, extract fields from a known schema, or run tests against a deterministic contract, use direct correctness checks. CORE-Bench, PaperBench, and MLE-bench demonstrate why verifiers and decomposed rubrics remain powerful when the endpoint can be specified.

Choose a narrower promotion state than “autonomous researcher”

After the audit, assign a use-specific state:

Most small teams should begin with execution_only or decision_support. A system can be excellent at literature search, data cleaning, experiment orchestration, and draft production without being trusted to decide that a product question is solved.

Agent Laboratory reported that human feedback between research stages improved its evaluated outputs. That is a project claim with its own methods and limits, not proof that any human-in-the-loop design works. The useful design principle is narrower: place human judgment at the forks where question framing, evidence sufficiency, and abandonment decisions can still change the work—not only after the report is finished.

Record a promotion receipt:

research_agent_promotion:
  system_version: "model + scaffold + tools + prompts"
  scenario: harbor-week-two-abandonment-v1
  runs_reviewed: null
  execution_score: null
  judgment_score: null
  zero_dimensions: []
  blocking_issues: []
  stable_failures: []
  allowed_state: "execution_only | decision_support | bounded_research | hold"
  allowed_domains: []
  forbidden_domains: []
  human_decision_owner: null
  expires_on: null
  evidence_bundle: null

Expire the receipt when the model, scaffold, memory policy, evidence tools, or review process changes materially. You are promoting a configured system for a defined job, not granting a permanent title to a model name.

A 48-hour Build Lab plan

Hours 0–4: choose one non-live decision, construct a safe evidence pack, write the frozen brief, and define execution and judgment scorecards. Keep the answer genuinely open but the authority narrow.

Hours 4–8: create the baseline run and six failure variants. Validate that every cited artifact has a stable locator. Name the human decision owner and reviewer.

Hours 8–28: run the configured agent. Capture system version, cost, wall-clock time, decision forks, tool trace, review artifacts, and final recommendation. Do not rescue the run silently; log every intervention.

Hours 28–36: have a reviewer score execution and judgment separately. Require evidence for every score. Compare the planned versus actual allocation of time and budget.

Hours 36–44: run the review-response test and at least one repeat of the most consequential failure case. Look for method changes, not only improved prose.

Hours 44–48: issue one narrow promotion receipt or hold the system. Preserve failed runs. They are the evidence that keeps the next successful report from becoming a selection-biased demo.

The change to make today is not to stop using research agents. It is to stop asking a finished report to prove a hidden capability. Search, code, experiments, charts, and prose show that an agent can execute a research workflow. A decision-fork ledger, disconfirmation run, and review-response diff show whether its choices deserve any authority.

References

1. CRUX — Can AI agents conduct open-ended AI research?
2. CRUX paper — Can AI agents conduct open-ended AI research?
3. CRUX in a Box — code, data, and available logs
4. METR — Evaluating frontier AI R&D capabilities against human experts (RE-Bench)
5. OpenAI — PaperBench
6. CORE-Bench repository
7. OpenAI — MLE-bench repository
8. Agent Laboratory: Using LLM Agents as Research Assistants
9. NeurIPS 2026 Reviewing Guidelines
10. NIST AI RMF Core — Measure
11. OpenAI — Scientific computing in the age of agentic AI