A model name is not a product version

The same model name can now describe different product behavior on the same day.

OpenAI’s August 6 update makes that unusually visible. The company says GPT-5.6 Sol in ChatGPT is being tuned for more focused answers, better factual reliability, and a more consistent experience between quick and deeper responses. It also says the updated version is only for the Chat experience: the GPT-5.6 Sol version powering Work and Codex is not changing in this release (official announcement).

That is not a naming mistake. It is a useful warning for every small team building with AI.

A label such as GPT-5.6 Sol, premium, or latest is not a complete experimental condition. The user-visible result also depends on the product surface, effective snapshot, reasoning setting, system instructions, tools, retrieval, safety layer, memory, account policy, and rollout cohort. If a team records only the model family, it may attribute a change to the wrong component, copy a result from one surface to another, or declare a regression without proving that the compared systems were equivalent.

OpenAI reports large factual-error reductions against GPT-5.5 Instant in an internal set of financial, medical, and legal prompts. Those are vendor measurements, not a guarantee for another product’s workload. We did not independently rerun them, and this article does not compare model quality. The artifact below is a proposed surface-change drill for teams to run on their own acceptance cases.

The immediate change is simple: replace “which model did we use?” with “which complete product surface produced this result?” Then freeze that surface, run paired fixtures, separate output quality from route and tool behavior, and promote only the exact configuration that earned the evidence.

What changed: one family name, at least four operating surfaces

The August 6 announcement describes a Chat-specific update. Plus and Pro users receive a new thought slider, and the announcement says the same GPT-5.6 Sol now powers Instant and deeper reasoning inside Chat. Free and Go users are moving to GPT-5.6 Luna with unlimited text chats and a separate Think control. Tool limits still apply.

Meanwhile, the current GPT-5.6 in ChatGPT guide still says GPT-5.5 Instant is the everyday default and GPT-5.6 Sol powers Medium and higher reasoning options. The two official pages therefore expose a documented routing conflict as observed on August 7. The newer announcement may describe a rollout the Help Center has not yet incorporated, or the labels may refer to different routing layers; the public pages do not resolve that question. This article does not choose one as the hidden production truth.

That conflict is the field note, not a footnote. A team that copied only the announcement would write “Sol” into its ledger. A team that copied only the Help Center might write “GPT-5.5 Instant.” Neither record proves which internal route served a particular answer. Both must retain the source URL, observation time, account context, visible controls, and unresolved status.

The same Help Center guide lists different availability for standard Chat, Work, Codex, and the API. Work and Codex can expose Sol, Terra, and Luna according to plan. The API exposes the family through model IDs and request parameters. These surfaces share branding, but they do not necessarily share the same effective instructions, tool contract, state model, rollout schedule, or safety behavior.

For a product team, distinguish at least four layers:

1. Family: the broad capability generation, such as GPT-5.6.
2. Tier or model: Sol, Terra, or Luna, including an API model ID or dated snapshot when available.
3. Surface: Chat, Work, Codex, API, or a third-party product built on one of them.
4. Configuration: effort, tools, system layer, memory, retrieval, account policy, client version, and rollout state.

The family name is useful for communication. It is too coarse for causal attribution.

OpenAI’s GPT-5.6 API model page makes the API boundary concrete. It separately documents aliases, snapshots, endpoints, tools, context limits, and rate limits. Its model guidance also tells developers to preserve a current reasoning setting as a baseline, compare another setting on representative tasks, and evaluate final task success rather than assuming a more capable route is always better (model guidance). None of those controls proves what happened in Chat, where the product owns more of the stack.

Define the experiment unit as a surface fingerprint

A useful surface fingerprint is the smallest record that lets a team understand which system was exercised. It is not necessarily enough for bit-for-bit reproduction; hosted products often do not expose every internal revision. Its job is to separate known values, provider-controlled values, and unknowns.

Record these fields before interpreting a result:

Do not fill unavailable fields with guesses. Write provider-controlled / not exposed. An unknown is evidence about reproducibility. It should not be disguised as a precise version.

This distinction also prevents a common category error: a UI label is an observation, while an API response field is a machine-readable claim. Neither establishes the provider’s entire internal deployment. Preserve exactly what the surface exposed and avoid inferring more.

Add one hard gate: if the team cannot establish that two runs share the same surface identity—or document exactly how they differ—it may report an experience difference but may not call it a model regression. That wording is deliberately restrictive. It stops a useful product observation from becoming a false causal claim.

Start with one product-shaped scenario, not a public benchmark

Suppose a small team ships an AI research assistant used to turn a short question into a decision brief. Customers can use an in-product API workflow, while the team also uses Chat and Codex internally to inspect failures and improve prompts.

The required brief has five product properties:

• answer the decision question in the first paragraph;
• cite every date, number, and rule near the claim;
• distinguish fact, vendor claim, inference, and unknown;
• preserve a defined JSON evidence object for downstream use;
• ask before any external write or purchase.

The August 6 Chat update sounds relevant because OpenAI claims tighter answers and fewer factual errors. But the team cannot paste ten examples into Chat, like the results, and assume its API product improved. Chat may use different instructions, web behavior, effort routing, source presentation, and rollout controls. Codex adds yet another tool and task environment.

Create a 24-case fixture instead. Use six real task shapes with private data removed or synthesized:

1. a current product-policy question with one recent date;
2. a pricing comparison with units and discount conditions;
3. a launch decision containing two conflicting sources;
4. a support question that should ask one clarification;
5. a dual-use request that should preserve a safe helpful path;
6. an action request where analysis is allowed but sending is not.

For each shape, create four difficulty variants: clean, ambiguous, source-missing, and adversarially distracting. Twenty-four cases are not a model leaderboard. They are small enough to review and broad enough to expose whether the product contract changed.

Build a surface-change manifest before running cases

Store the fingerprint and fixture revision in one record. The manifest prevents the team from silently changing the prompt or tool set midway through the experiment.

surface_change_manifest:
  experiment_id: "decision-brief-surface-2026-08"
  hypothesis: "A candidate surface improves focused, cited briefs without losing required fields or approval boundaries."
  fixture:
    version: "brief-suite-v3"
    case_count: 24
  baseline:
    surface: "pin-current-production-surface"
    account_plan: "record"
    client_version: "record"
    observed_at: null
    visible_control: "record"
    documented_route:
      claim: "record"
      source_url: "record"
      conflict_status: "none | unresolved"
    displayed_model: "record"
    requested_model_id: "record-if-api"
    resolved_snapshot: "record-if-exposed"
    reasoning_control: "record"
    prompt_revision: "sha256-or-version"
    tools_revision: "sha256-or-version"
  candidate:
    surface: "record"
    account_plan: "record"
    client_version: "record"
    observed_at: null
    visible_control: "record"
    documented_route:
      claim: "record"
      source_url: "record"
      conflict_status: "none | unresolved"
    displayed_model: "record"
    requested_model_id: "record-if-api"
    resolved_snapshot: "record-if-exposed"
    reasoning_control: "record"
    prompt_revision: "sha256-or-version"
    tools_revision: "sha256-or-version"
  unknowns: []

Hash or version tool schemas as well as prompts. A model with file search and a model without it are not the same treatment. Neither are two tool implementations that share a name but return different fields.

If a hosted UI does not expose a prompt or tool revision, write that limitation and avoid claiming a controlled model-only comparison. The experiment may still answer a product question—“which surface is better for our reviewer?”—without answering a scientific one—“which weights are better?”

Cross the surfaces without pretending they are interchangeable

Use a matrix that keeps the decision attached to its intended job:

This is the central discipline. Cross-surface comparisons can be useful, but only if the conclusion is also cross-surface. They cannot isolate a model revision because the harness changed with the surface.

If documented_route.conflict_status remains unresolved, the first two matrix rows need different language. “Chat answers became shorter on these cases after the visible August rollout” is supportable. “GPT-5.6 Sol became shorter” is not, unless the serving identity is established independently. The subject of the conclusion must match the identity the experiment actually observed.

Run each case at least three times when sampling or hidden routing can vary. Randomize case order. Start new conversations or tasks unless state is deliberately part of the condition. Record discarded runs and infrastructure errors rather than replacing them invisibly.

If a UI has a thought slider, record its visible position and product label. Do not translate it into an API reasoning.effort value unless the provider explicitly documents that mapping. Similar controls are not proof of equal budgets or orchestration.

Score the contract in layers

A single “quality” score hides why the result changed. Keep five score families separate.

1. Surface identity. Did the run preserve the account, client, visible control, timestamp, displayed route, documentation source, and conflict status? A missing identity blocks model-level attribution even if the answer looks better.

2. Contract completion. Did the response answer first, include the required sections, produce valid structured data, and respect the requested length? These checks can often be deterministic.

3. Evidence quality. Are material facts supported? Do cited pages actually contain the claim? Are dates, quantities, units, and policy boundaries correct? Human review is usually required for the most consequential claims.

4. Interaction behavior. Did the system ask a needed clarification, avoid repetitive restatement, preserve context, and hand control back at the right moment? This is where a Chat-specific tuning update may matter even when the underlying task facts do not.

5. Decision usefulness and repair cost. Does the output identify the real trade-off, state uncertainty, and give a usable next action? Record reviewer correction minutes, rejected outputs, latency, tokens where available, tool calls, and retries. UI surfaces may hide token data; mark it unavailable instead of inventing a comparison.

OpenAI’s evaluation best-practices guide recommends task-specific evaluations, logging, continuous evaluation, human calibration, and comparison rather than vague open-ended scoring. The older but still useful ML Test Score similarly treats data, model, infrastructure, and monitoring tests as parts of production readiness. The practical lesson is not to imitate a large platform. It is to make the score explain which contract layer moved.

Separate confirmed facts, vendor claims, inferences, and unknowns

Use four evidence labels in the decision note.

Confirmed product fact: OpenAI says the August 6 GPT-5.6 Sol update is limited to Chat and does not change the Work/Codex version in that release. This is a direct scope statement.

Vendor claim: OpenAI reports that responses with at least one factual error were about 68% less common for GPT-5.6 Sol than GPT-5.5 Instant in its internal financial, medical, and legal evaluation. The prompt set, adjudication detail, and full result distribution are not provided on the announcement page, so the number cannot be transferred to a product’s acceptance rate.

Author inference: Same-name surfaces should be versioned as distinct experimental conditions. That conclusion follows from the documented scope split and the separate API configuration surface, but the proposed manifest is Y Build’s method, not an OpenAI standard.

Unknown: Whether a particular account has received the rollout, which internal revision produced a single Chat answer, and whether Chat’s visible thought control maps to an API effort value. The public pages do not establish those details.

This evidence grammar reduces accidental marketing language. It also makes later review easier: a newly published snapshot ID may resolve an unknown without changing the rest of the experiment.

Failure modes that can produce a false regression

The rollout cohort changed. Baseline and candidate were run on different accounts, plans, regions, clients, or dates. The team attributes an entitlement or rollout difference to model quality.

The surface changed twice. A model update and a prompt, retrieval, tool, or safety-policy change land together. The output moves, but the experiment cannot identify the cause.

A floating alias is treated as a pin. The API request records a family alias but not a resolved snapshot or run time. Later reruns may exercise a different backend.

The judge moves with the candidate. The same changing model or surface grades its own new output. A rubric revision is also applied mid-run, making old and new scores incomparable.

State leaks across cases. Chat history, project memory, retained reasoning, cached retrieval, or prior files influence later cases. The apparent improvement is context carryover.

Missing telemetry becomes zero. Token counts, tool failures, citations, or dropped runs are unavailable on one surface. The dashboard treats missing values as better performance.

Averages hide a critical slice. Overall style improves while source-missing or action-boundary cases regress. Promotion follows the mean and ignores a blocking invariant.

The team copies a Chat result into an API claim. A person observes better everyday answers and assumes a production endpoint has changed. OpenAI’s own scope note shows why that inference is unsafe here.

The Microsoft study on experimentation under telemetry loss is not about language-model surfaces specifically, but it supports the broader measurement warning: missing outcome telemetry can bias an experiment and its decision. Treat unknown as a state that blocks some conclusions, not as a convenient zero.

Promotion requires a configuration-specific receipt

Do not promote “GPT-5.6.” Promote one complete route.

surface_change_decision:
  experiment_id: "decision-brief-surface-2026-08"
  decision: "promote | hold | reject"
  decision_subject: "observed_surface | api_configuration | model_revision"
  causal_attribution: "available | unavailable"
  promoted_scope:
    product_job: "decision-brief generation"
    surface: "api"
    requested_model_id: "pin"
    resolved_snapshot: "pin-if-exposed"
    reasoning_control: "pin"
    prompt_revision: "pin"
    tools_revision: "pin"
  evidence:
    surface_identity_complete: false
    documentation_conflicts: []
    completed_case_runs: 0
    contract_pass_rate: null
    critical_unsupported_claims: null
    reviewer_correction_minutes_p50: null
    latency_p95_ms: null
    cost_per_accepted_brief: null
  blockers:
    invalid_json: 0
    unauthorized_external_actions: 0
    missing_required_approval: 0
    unresolved_critical_source_disagreements: 0
  rollback_trigger: "define"
  approved_by: []
  remaining_unknowns: []

Require zero blocking invariant failures before promotion. Enforce this rule in the receipt validator: when surface_identity_complete: false, causal_attribution must be unavailable, and decision_subject must not be model_revision. The team may still set decision: promote for an observed workflow surface that works better, but it is promoting that surface for the named job—not GPT-5.6 as a model. For softer metrics, define a margin before the run. A 1% style preference should not outweigh a 20% increase in correction time, and an unmeasured cost field should not be reported as unchanged.

Roll out the API route gradually if production traffic is in scope. Keep the baseline route available. Monitor the same task slices used in the fixture and preserve a rollback trigger. For internal Chat, Work, or Codex workflows, update the playbook with the observed account, client, date, and allowed task—not a universal claim about the model family.

When this protocol is useful—and when it is not

Use the surface-change ledger when a provider updates a named model, a UI changes routing or reasoning controls, an alias moves, a managed workspace changes policy, or the same family appears in several products. It is especially valuable when teams use a consumer surface for manual work and an API surface for customer-facing automation.

Do not use a 24-case fixture to claim general intelligence rankings, medical or legal safety, population-level fairness, or provider-wide superiority. Those require larger, domain-qualified evaluations and more independent evidence. Do not use synthetic cases as a substitute for post-launch monitoring. Do not log real sensitive prompts merely to improve reproducibility; minimize, redact, or construct safe equivalents.

The NIST AI Risk Management Framework emphasizes that measurement and risk treatment must fit the context of use. A surface ledger follows that principle by attaching evidence to a product job and a configured system. It does not certify the provider, the family, or every downstream use.

This protocol is also unnecessary for a purely deterministic change with complete unit coverage. If no model or hosted product behavior participates, ordinary version control and tests are the better tool.

A 48-hour Build Lab sequence

Hours 0–2: freeze the evidence. Write each vendor statement with source and observation time, the intended product job, baseline route, candidate route, and allowed conclusion.

Hours 2–6: assemble the fixture. Select six task shapes, create four difficulty variants, remove customer data, freeze sources, and define blocking invariants.

Hours 6–12: fingerprint the surfaces. Capture account, client, visible control, documented route, model label or ID, prompt, tools, state policy, timestamp, and rollout indicators. Hash what the team controls.

Hours 12–24: run paired cases. Randomize order, reset state, repeat each case three times where needed, and retain errors and discarded runs.

Hours 24–36: review blind and inspect the weakest slice. Score identity, contract completion, evidence, interaction, usefulness, and repair cost. Compare ambiguous, source-missing, action-boundary, and long-context cases separately.

Hours 36–42: write the receipt. Promote, hold, or reject one configuration. List the claim boundary and remaining unknowns.

Hours 42–48: stage and watch. Use a small rollout or an internal-playbook change, set rollback triggers, and schedule the next review.

The purpose is not to make a small team imitate a model lab. It is to stop a product label from becoming fake precision. When a provider says one GPT-5.6 Sol surface changed and another did not, the right response is not confusion. It is a better experiment unit.

References

1. OpenAI, “Improving GPT-5.6 Sol in ChatGPT—and expanding access to GPT-5.6 Luna for free users”
2. OpenAI Help Center, “GPT-5.6 in ChatGPT”
3. OpenAI API, “GPT-5.6 Sol Model”
4. OpenAI API, “Model guidance”
5. OpenAI API, “Evaluation best practices”
6. NIST, “AI Risk Management Framework”
7. Google Research, “The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction”
8. Microsoft Research, “Trustworthy Experimentation Under Telemetry Loss”