Your AI builder changes an export screen. The download opens, the requested rows are present, and the customer can use the file. The evaluation still marks the task wrong because it expected a particular filename. Elsewhere, a superficially similar change passes even though it exports records from another workspace.
Replacing the model might change both scores. It would leave the team unable to explain either result.
Before buying another round of model comparisons, inspect the two contracts surrounding the work: what information the builder received, and what the evaluator treated as success. A clear request can meet a bad test. A reasonable test can be attached to an incomplete request. The fixes are different, and combining them into one improved score hides the difference.
This field note proposes a small, unexecuted study for an AI product team’s own task queue. It uses independent cold reads, deliberately varied submissions, and two separate comparisons. First, rescore identical saved outputs to audit the grader. Then, if useful, run a fresh clarification experiment with one fixed acceptance standard. There are no model runs, customer observations, or measured improvements behind the examples below.
What the new research puts on the agenda
The October 29 ExLabel / SWE-TQ paper, discovered in today’s briefing, assesses two dimensions: whether a coding task provides sufficient information and whether its tests can validate solutions fairly. SWE-TQ spans eight programming languages. ExLabel reuses an evolving bank of assessment experience. Its authors report stronger annotation performance, but that is not a product team’s repair success rate. The study retains unanimous source annotations and applies further cleaning; its labels are not an exhaustive inventory of disagreement.
The released repository provides code and data, while its README says the experience banks must be built separately. We have not run that pipeline. For a small team, the immediate opportunity is a narrower question: when a task fails, did the builder lack a stated requirement, did the grader impose one, or did the implementation actually fail?
That question predates this paper. OpenAI’s SWE-bench Verified account documents an issue whose hidden test requires warning text absent from the supplied description. Its human screening examines both descriptions and tests. This older work supplies a concrete precedent, not evidence that today’s application has the same defect rate.
Separate the request, the contract, and the grader
For this experiment, the request packet is everything the builder may use: the task text, permitted files, visible examples, relevant interface, and allowed documentation. If users can ask follow-up questions, the permitted interaction belongs in the packet too. A hidden requirement cannot be made visible by saying that a competent engineer would have known it.
The acceptance contract describes the behaviors the product actually needs. It can allow several implementations or outputs. It must distinguish required behavior from preference, and both from unresolved product decisions. The grader is the mechanism that checks a submission against that contract: code, human review, or a combination.
These definitions create useful diagnostic disagreements. A grader can correctly reject a missing required behavior that the request never communicated. That is a request defect even when the rejection is mechanically accurate. Conversely, a builder may satisfy every stated behavior and lose points for an evaluator’s private formatting preference. That is a grader defect.
The SWE-bench documentation describes an issue-plus-codebase task that produces a patch. A product workflow often exposes more context than that. Write down your actual information boundary rather than borrowing a benchmark’s restrictions. A link is usable evidence only when the environment permits opening it and the relevant version is available.
Start with one export change, not the entire backlog
Consider a fictional team building a workspace-based service called ParcelDesk. Its task says: “Add CSV export of the current orders.” The evaluator expects orders.csv, the current workspace only, the selected filters, a particular column order, and a stable interpretation of the displayed date range.
Those are not equally justified expectations. Workspace isolation is a product requirement that must already exist in the accessible product contract. “Current” may refer to the current page, the complete filtered result, or all orders. A filename may matter to a downstream integration, or it may be a test author’s preference. The team must settle these questions before interpreting a model result.
For the study, the product owner chooses the complete filtered result in the current workspace, fixes the date-range semantics and schema, and explicitly declares filename spelling irrelevant. This is a designed example, not a recommendation for every export product. If an existing integration requires an exact filename, keep that constraint and disclose it.
Sample a modest set of tasks from one workflow family, such as export changes. Keep the sampling rule and the original wording. Include routine requests, known disputes, and cases suspected to be clear, but report these groups separately. A dispute-heavy sample is useful for diagnosis; its defect frequency cannot stand in for your whole backlog.
Use cold reads to find missing decisions
Ask two reviewers who did not author the task to read the same packet independently. They should have enough domain knowledge to recognize normal product requirements, but neither should see the hidden grader, reference solution, model identity, or the other reviewer’s interpretation.
Each reviewer writes the observable outcome they would deliver, two plausible edge cases, and every point that requires a product decision. They must cite where each required behavior appears in the packet. “Everyone knows this” is a finding to investigate, not a source location.
Then compare their interpretations. Two different filenames can both be acceptable. Different workspace boundaries cannot both be accepted if the product promises isolation. Do not count every wording difference as ambiguity. Record a disagreement only when it could change the implementation, the customer outcome, or the verdict.
The owner adjudicates against documented product intent and can mark an issue unresolved. Agreement between two reviewers is a diagnostic check, not proof of clarity; they may share the same unstated assumption. Add a third, less familiar reader when that risk matters. Preserve the original independent notes before discussion, so consensus does not erase the ambiguity that triggered it.
The Multi-SWE-bench repository records a later addition of task hints describing newly defined variables. That is a reminder to version information exposure. If a repair supplies a formerly hidden identifier, the next run receives a different task packet, even when the underlying customer request is unchanged.
Challenge the grader with valid alternatives and deliberate faults
Before running another agent, construct a small set of submissions whose intended status the product owner and an independent reviewer agree on. Use a disposable fixture with invented records. Never seed cross-workspace faults in a live customer export.
Include at least two behaviorally valid alternatives and several faults that violate different requirements. In ParcelDesk, valid alternatives might stream the same permitted CSV rows or generate the file after collecting them. Test both against the same frozen data and schema. Their internal architecture should not decide acceptance unless performance or another relevant constraint explicitly makes it part of the contract.
Google’s older test-behavior guidance explains why implementation-coupled assertions can break legitimate changes. It does not mean all exact assertions are wrong. CSV headers, protocol keys, and legally required wording may be exact contracts. The audit asks whether the assertion has a defensible requirement behind it.
| Designed submission | Contract verdict | What a disagreement reveals |
|---|---|---|
| Correct permitted rows and schema, different allowed filename | Accept | Filename preference may have become a hidden requirement |
| Same required result through a different valid implementation | Accept | Grader may depend on implementation details |
| Exports only the visible page although the contract requires all filtered rows | Reject | Happy-path fixture may not expose truncation |
| Includes an invented record from a second workspace | Reject | Isolation check may be missing |
| Has correct headers but drops a required field value | Reject | Structure-only checking may accept incomplete data |
| Date-boundary inclusion violates the frozen date rule | Reject | Fixture may not exercise the boundary |
These are proposed challenge cases, not observed results. A reviewer must verify each submission’s label independently of the grader under audit. If the reviewers cannot agree, quarantine the case and retain its disputed status; do not quietly use it as ground truth.
For the export fixture, prepare a separate expected-result manifest from the invented input records and the frozen product rules. List the permitted row IDs, required cell values, excluded workspace IDs, and boundary-date decisions. Have a second reviewer check that manifest directly against the inputs before it is used to label submissions. Copying the current grader’s expected output would reproduce the very assumptions this exercise is meant to inspect.
Keep the initial labels hidden from each other and from the grader verdicts until both reviewers have finished. Record the evidence for each acceptance or rejection, including the exact mismatching row or value. When a disagreement exposes a flaw in the manifest, version the correction and revisit every affected label. When it exposes an unsettled requirement, return to the product owner instead of voting the case into the accepted set. This extra work is why a few carefully labelled counterexamples are more useful here than hundreds of automatically generated submissions with uncertain status.
Repair scoring without pretending the builder improved
The first comparison holds submissions constant. Save their bytes, fixture state, relevant execution records, and contract version. Run the original grader and the proposed repaired grader on those same artifacts. If a grader needs executable behavior, replay each submission in a fresh copy of the same fixture rather than sharing mutable state between evaluations.
Record every verdict change in both directions. A valid alternative newly accepted is a corrected rejection. A deliberately faulty submission newly rejected is a corrected acceptance. An unchanged verdict can still be wrong; compare both graders with the independently adjudicated label, not just with each other.
Report counts for the valid and faulty sets separately. For example, use blank fields for “valid alternatives rejected / adjudicated valid alternatives” and “faulty submissions accepted / adjudicated faulty submissions.” Do not populate those fields with imagined rates. Keep disputed cases outside these labelled denominators and report their count alongside them.
This stage estimates agreement with a small challenge set. It does not estimate production accuracy, and it does not show that the builder learned anything. Rescoring old outputs may change a dashboard immediately while changing no delivered work. Keep that change visible as a measurement revision, with the old score and grader version still retrievable.
Test clarification in a separate, fresh run
Only after the acceptance contract and audit grader are frozen should the team test whether a clearer request changes behavior. Compare the original packet with a repaired packet that supplies the missing product decisions. Use the same model version, tools, initial repository, data, budget, and interaction policy in both conditions.
Start each attempt in fresh state and randomize execution order. Do not let the original run’s transcript, generated patch, reviewer notes, or challenge submissions enter the clarified run. If follow-up questions are allowed, apply the same rule and record the answers. Otherwise you are comparing different interaction services rather than wording alone.
Judge both conditions against one fixed acceptance contract with condition labels hidden from the reviewer. Preserve task-level pairs and repeat stochastic attempts if your budget permits. Count every scheduled attempt: accepted behavior, rejected behavior, no submission, and execution failure must remain distinguishable. An environment failure is not a demonstrated implementation defect, but it still consumes part of the planned study and needs a disclosed handling rule.
A six-task pilot can reveal concrete misunderstandings; it cannot establish a reliable population effect or model ranking. Predeclare the pilot size, repetitions, stopping rule, and primary outcome before execution. If an essential contract decision changes halfway through, stop the comparison and start a new version. Do not splice the two versions into one success rate.
Keep one repair sheet per task
The following sheet is the reusable artifact. Complete one sheet for each task, linking out to immutable packets and evidence rather than pasting confidential traces into a general spreadsheet. The columns distinguish an input repair, a scoring repair, and a product decision that has not yet been made.
| Field | What to record before or during the study |
|---|---|
| Task and sample group | Stable ID, workflow family, routine/disputed/clear-control selection reason |
| Original packet | Text and accessible context versions; allowed links, tools, and questions |
| Required behavior | Observable outcome, exclusions, and the source of each requirement |
| Open decision | Specific ambiguity; owner; unresolved or adjudicated status |
| Independent cold reads | Two original interpretations and decision-changing disagreements |
| Valid alternatives | At least two adjudicated examples and why both satisfy the contract |
| Deliberate faults | Requirement violated, expected rejection, independent label evidence |
| Frozen acceptance contract | Version, owner sign-off, allowed variation, disputed-case register |
| Grader-only comparison | Same artifacts, old/new grader versions, both verdicts, execution status |
| Clarification comparison | Original/repaired packet IDs, fresh run IDs, randomized order, fixed grader |
| Results and exclusions | Separate denominators, missing outputs, execution failures, unresolved labels |
| Decision and next check | Repair input, repair grader, investigate implementation, or defer; new validation set |
The sheet should let someone reconstruct why a verdict changed without asking the task author to remember a meeting. Store the requirements behind the label, not just an “approved” checkbox. Where data access is sensitive, record a controlled location and authorized reviewer rather than copying the data.
Keep reference submissions out of future builder inputs. Once a task has been discussed or used to tune prompts, it becomes development material. A repaired grader should meet a fresh challenge set before it becomes a release authority. That new set needs valid variation and relevant faults too; a pile of known bad examples alone cannot detect unfair rejection.
Watch for repairs that make the evaluation easier to game
The obvious failure is replacing an exact filename check with “a download exists.” That repairs one unfair rejection by accepting almost anything. Every relaxed assertion needs a behavioral replacement that still protects the requirement. Inspect the rows, their workspace ownership, their completeness, and the agreed schema; do not merely check that a CSV parser opens the file.
Mutation testing supplies a useful engineering analogy. PIT’s documentation describes altering code and checking whether tests detect the change. Here, deliberately faulty submissions ask whether acceptance checks notice a meaningful behavioral defect. This manual challenge set is not a PIT run, a mutation score, or exhaustive fault coverage. Equivalent changes that preserve required behavior should not be counted as defects.
Likewise, Hypothesis explains testing properties over generated inputs. A team implementing the export audit could encode a property that adding an out-of-workspace record leaves the permitted export unchanged. Such a property needs an independently justified scope and fixture; generated cases cannot tell you whether “current orders” meant the visible page in the first place.
Another trap is clarifying the task by revealing the answer. “Use this exact patch” tests copying, not implementation. State the required behavior and necessary interface facts while preserving legitimate solution choices. Record every added clue so a later reader can see whether you removed ambiguity or substantially reduced the problem.
Treat disagreement as work, not dirty data
Do not delete disputed tasks to make the evaluation look stable. A small team can maintain three states: usable for a frozen comparison, needs a product decision, and invalid under the current environment. Preserve movement between states and the reason for it. Report how many tasks leave the sample, and whether the removed tasks represent important customer situations.
Reviewer independence also has limits. Two people may consult the same misleading internal document. A product owner may unconsciously defend the test they wrote. Hide provider identity and current scores, require citations to requirements, and invite a reviewer to present a valid counterexample. These practices expose disagreement; they do not guarantee an objective oracle.
The method fits bounded product changes with observable outcomes: exports, imports, filters, permission checks, and small workflow edits. It is less decisive for taste-heavy design, exploratory research, or jobs whose customer value cannot be specified in advance. In those settings, cold reads can still improve a brief, but a binary grader may misrepresent the work.
Do not turn uncertainty into permission to relax privacy, security, or contractual commitments. When an essential boundary lacks a clear specification or trustworthy check, defer the affected release. Resolving the requirement is product work; a different model cannot make that responsibility disappear.
Make the next decision narrower than “switch models”
The useful outcome is a diagnosis. If the grader rejects valid alternatives or accepts meaningful faults, repair the measurement and validate it on fresh examples. If cold reads reveal missing decisions, repair the packet and then test continuation under the fixed contract. If a clear packet and defensible grader still expose implementation failures, investigate the builder, tools, or model.
Track reviewer time, task-author clarification time, replay cost, and unresolved decisions alongside the results. A more elaborate task packet can become expensive to maintain or difficult for users to supply. The next product experiment may be a short clarification interaction, not a longer permanent prompt. That interaction deserves its own study rather than being hidden inside a model comparison.
Start with one disputed task and one apparently clear task from the same workflow. Preserve their original packets today. Ask for independent interpretations, challenge their graders, and decide which comparison is warranted. Expand only when the procedure produces evidence you can inspect. The goal is to stop spending model budget on a question that your current task and test cannot answer.
References
- Jiang et al., Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent, October 29, 2026. Primary research; author-reported experiments.
- ZJU-CTAG, ExLabel repository. Released code, data, configuration, and reproduction instructions; not executed for this article.
- OpenAI, Introducing SWE-bench Verified. Primary annotation-method account, originally August 2024; historical background.
- SWE-bench authors, SWE-bench documentation. Task and evaluation context.
- Multi-SWE-bench authors, Multi-SWE-bench repository. Task information and harness context; historical hints update.
- Andrew Trenk / Google Testing Blog, Test Behavior, Not Implementation, August 2013. Older engineering guidance.
- PIT maintainers, PIT Mutation Testing. Fault-seeding method background; no tool run claimed.
- Hypothesis maintainers, Hypothesis documentation. Property-based testing background; no generated-input experiment claimed.