OpenRouter Classifiers can label AI spend. First prove the labels

OpenRouter released Classifiers in beta on July 24, 2026. A workspace can define a taxonomy, choose a model, and ask that model to tag every generation—or a sample—with fields such as department, task type, audience, agent complexity, or potential capitalizable software expense. Those tags can then appear beside requests and roll up into activity reporting.

That is useful. It is also easy to misuse.

A structured label is not an accounting fact just because it fits a schema. The classifier sees a serialized, truncated view of the conversation. It can miss a job, force an ambiguous request into the wrong bucket, fail silently, or become less representative when a team samples traffic. If the resulting chart says “Marketing consumed 38% of AI spend,” the number combines metered cost with a model-generated judgment about what the work meant.

For a small team, the decision is not whether semantic cost attribution looks convenient. It is whether the labels are reliable enough for the action attached to them. A rough product-usage trend needs less evidence than a budget transfer, client invoice, compliance review, or capitalization decision.

This field note proposes a calibration experiment, not a reported OpenRouter benchmark. Y Build did not run Classifiers against private workspace traffic, and no result below is presented as observed performance. The reusable artifacts are a taxonomy contract, a 120-record adjudication ledger, coverage and error gates, a sampling check, a shadow-run protocol, and a promotion receipt. The goal is to let semantic tags inform decisions without quietly becoming a second, less-auditable billing system.

What OpenRouter launched—and what it did not prove

OpenRouter’s launch announcement describes four configuration parts: a taxonomy of up to eight dimensions, an instruction prompt, a classification model, and a sampling rate. Six presets include department, audience, task type, engineering work, agent complexity, and potential capitalizable software expense. Classification runs after the primary generation, so it does not add latency to the user-facing inference path.

The detailed Classifiers documentation adds the operational boundaries that matter for a release decision:

• the prompt is serialized as a labeled transcript containing system text, tool names, user and assistant turns, tool calls, and tool results;
• full tool schemas are not included;
• each turn is truncated to at most 5,000 characters;
• a classifier with a meaningfully shorter context than the original prompt can fail silently;
• timeouts, model errors, and invalid outputs leave the original generation intact but write no tags;
• classifier calls consume workspace credits like other generations;
• the configured administrative user, rather than the original API key, is billed for the classifier request.

OpenRouter also says its structured-output constraint limits results to declared dimensions and values. That proves the output can be shaped. It does not prove the selected value is correct, that missing tags are random, that sampled traffic represents total traffic, or that a label is suitable for financial reporting.

The product is beta. The launch is enough reason to run a bounded evaluation. It is not evidence that a preset taxonomy matches a particular company or that one recommended low-cost model will meet every team’s error budget.

Separate metered cost from inferred meaning

An AI usage ledger has at least two different kinds of fields.

Metered fields describe what the platform recorded: generation ID, model, prompt tokens, completion tokens, cache use, and cost. OpenRouter’s usage-accounting documentation says these details are returned with each response and can also be retrieved by generation ID. They are still subject to the platform’s billing definitions, but they are not semantic guesses about the work.

Inferred fields interpret the request: department, task type, audience, project, difficulty, or accounting treatment. A model assigns them from the transcript it receives. They require a taxonomy, evidence, and a policy for uncertainty.

Keep the two layers separate in storage and in language:

This distinction prevents a common leap: classifier_label = accounting_owner. A salesperson can ask for code, an engineer can draft a launch page, and a shared agent can perform one workflow for several departments. Even a correct task label may not answer who owns the budget.

Treat semantic classification as a join candidate, not the final ledger. Cost allocation should combine metered usage with declared ownership, reviewed meaning where necessary, and an explicit allocation policy.

Write a taxonomy contract before evaluating the model

Taxonomy errors often look like model errors. If “Operations” overlaps with “Customer Support,” or “Bug fixing” overlaps with “Incident response,” reasonable reviewers will disagree before a classifier is involved.

Create a contract for every dimension:

Use examples drawn from the team’s own jobs, but sanitize customer data and secrets. Include boundary examples, not only easy prototypes. “Draft a pricing email” is an easy Marketing example. “Analyze churn, write SQL, and draft a renewal message” reveals whether the taxonomy describes the requesting department, the work performed, the artifact produced, or the beneficiary.

Add unknown, mixed, or an explicit escalation rule where the product permits it. A closed schema without an uncertainty path does not remove ambiguity; it converts ambiguity into false certainty.

OpenRouter’s capitalizable-software preset is a particularly important boundary. Its own documentation warns that customers remain responsible for final financial or tax data. A classifier may help retrieve potentially relevant work. It should not decide capitalization policy, useful life, or audit treatment on its own.

Build a 120-record adjudication ledger

The first useful experiment does not require classifying the whole workspace. Build a proposed 120-record set from permission-safe historical patterns or synthetic equivalents.

Stratify it deliberately:

• 40 common, single-purpose requests;
• 20 multi-turn requests whose purpose changes;
• 15 tool-heavy agent runs;
• 15 long turns that will exercise truncation;
• 10 mixed-department or mixed-project jobs;
• 10 requests that should become unknown;
• 10 high-consequence records tied to finance, compliance, client billing, or sensitive data.

The counts are a starting fixture, not a statistical guarantee. Increase them when labels are rare or actions are consequential.

Two reviewers should label a subset independently using the taxonomy contract. Reviewer agreement is not the same as truth, but disagreement reveals unclear definitions. A meta-analysis in Computational Linguistics found that annotation purpose, training, category design, and other process choices materially affect reported agreement; the practical lesson is to document the annotation system rather than quote one universal agreement threshold (Bayerl and Paul).

Resolve disagreements through adjudication, recording the reason instead of overwriting the first labels. The ledger should retain:

record_id
source_pattern
risk_tier
taxonomy_version
reviewer_a_label
reviewer_b_label
agreement
adjudicated_label
adjudication_reason
evidence_available_to_classifier
truncation_expected
classifier_model
classifier_prompt_version
classifier_label
classifier_status
metered_classifier_cost

Do not use the classifier to generate its own gold labels. Do not let reviewers see the model’s label before independent annotation. Otherwise the experiment measures agreement with a suggestion, not independent validity.

Gate on coverage before calculating accuracy

Classification can fail without breaking the original request. That is a sensible product choice for inference availability, but it makes coverage a first-class metric.

Define:

eligible_records =
  generations the policy expected to classify

tagged_records =
  eligible records with a complete, valid label set

coverage =
  tagged_records / eligible_records

Then split missing records by cause where the platform exposes enough evidence: timeout, invalid output, insufficient context, administrative configuration, model availability, or unknown. If the cause is not exposed, record it as unknown rather than inventing a diagnosis.

Coverage must also be sliced by meaningful cohorts:

• long versus short turns;
• tool-heavy versus text-only work;
• language;
• department or API key;
• classifier model;
• taxonomy version;
• ordinary versus high-consequence records.

An overall 96% coverage rate can hide 60% coverage on long compliance conversations. Missing labels are not harmless if the missingness correlates with costly or complex jobs.

Google’s production ML guidance argues that tests and monitoring belong alongside offline model metrics; its ML Test Score turns that principle into concrete production-readiness checks. For this feature, the release gate begins with “Did the classification job produce auditable output for the records we expected?” before asking whether the output was correct.

Use a confusion matrix, then price the mistakes

For every tagged record in the adjudicated set, compare the classifier label with the reviewed label. A multi-class confusion matrix shows which categories absorb one another.

Accuracy alone is weak when categories are imbalanced. Google’s classification metrics guide explains why a system can look accurate by predicting the dominant class while missing rare, important cases. Report per-label precision and recall:

• Precision: when the classifier assigns Legal, how often is the adjudicated label Legal?
• Recall: of all adjudicated Legal records, how many did the classifier find?
• Unknown precision: when the classifier abstains, how often was evidence genuinely insufficient?
• Unknown recall: of ambiguous records, how often did the classifier avoid a forced label?

Then attach error cost. Confusing Documentation with Feature Development may distort an internal trend. Confusing client work with internal work can affect an invoice. Marking maintenance as potentially capitalizable can trigger a faulty finance workflow if nobody reviews it.

Use an error table:

Keep the count cells blank until a real run produces evidence. The weighted score is a team policy, not an objective property of the classifier. Never let many low-cost correct labels cancel one critical error.

Audit sampling before extrapolating totals

Sampling controls classifier cost, but the sampling design determines which conclusions are supportable. Ten percent of requests is not automatically ten percent of spend. Large agent runs may be rare and expensive; interactive chat may be frequent and cheap.

OpenRouter’s public State of AI report provides a useful boundary example: it calls its dataset observational and says its internal categorization uses a random sample of roughly 0.25% of prompts. That report is not evidence about a customer’s Custom Classifiers configuration. It demonstrates the disclosure a sampled analysis needs: what was sampled, how the labels were produced, and which population the result describes.

For a workspace experiment:

1. Generate a stable random selection using a recorded rule where possible.
2. Compare sampled and unsampled traffic on fields available without semantic labels: API key, model, token volume, cost, latency, time of day, and tool use.
3. Oversample rare high-consequence cohorts and retain their selection probabilities.
4. Report observed sample results separately from weighted estimates.
5. Keep raw metered total cost outside the semantic classifier; only the allocation proportions are inferred.

If sampling is configurable only as a percentage without a reproducible seed or exportable inclusion record, treat fine-grained extrapolation as limited. You can still use the tags for directional exploration. Do not present a precise departmental dollar amount unless you can explain how records entered the sample and how missing classifications were handled.

Run in shadow before labels move money

The first production stage should write tags without changing budgets, model routes, invoices, permissions, or finance records.

Run the proposed shadow phase for seven days or a full business cycle, whichever captures the team’s actual variation. Preserve the taxonomy version, classifier model, classifier prompt version, sample rate, and activation time. Each day:

• reconcile eligible, sampled, tagged, failed, and unclassified counts;
• review all high-consequence labels;
• review a random slice of ordinary labels;
• inspect disagreement clusters rather than only aggregate accuracy;
• calculate classification cost as a share of the inference cost being analyzed;
• record taxonomy edits as new versions, not silent prompt changes.

OpenRouter’s Activity export reports spend, tokens, and requests and can group by model, API key, or creator. Use those metered groupings as reconciliation anchors. The semantic totals should add back to the tagged portion of the same observed cost window. Differences need an explicit bucket: unsampled, failed, unknown, late-arriving, or excluded by policy.

NIST’s AI RMF Measure function calls for documented testing, performance assessment, uncertainty, production monitoring, and independent review (AI RMF Core). The principle here is simple: approval expires. Recheck a sentinel set when the taxonomy, classifier prompt, model, context handling, product workflow, or traffic mix changes.

A concrete scenario: one agent, three cost centers

Consider a hypothetical 12-person software company using an agent for support, onboarding, and product operations. One weekly workflow:

1. reads support tickets;
2. queries product usage;
3. drafts a help article;
4. proposes an onboarding email;
5. opens an engineering issue.

Which department owns the cost? The answer depends on the company’s allocation policy. The requestor may be Support. The largest token share may come from Product analysis. The final artifacts may benefit Marketing and Engineering. A department classifier can be semantically reasonable and still be wrong for the budget question.

Run three separate dimensions instead:

• requesting_owner: declared from workspace or API-key metadata;
• work_type: inferred from the transcript;
• allocation_policy: applied after review, possibly with shared or split percentages.

The classifier is best positioned for work_type. Keep requesting_owner declared. Keep allocation_policy in a versioned rule owned by a person.

The experiment should include workflows that change purpose mid-conversation, because the serialized transcript may contain several jobs. Decide whether the label describes the latest request, dominant work, final artifact, or entire generation. If the team cannot state that rule, a chart built from the labels will answer a question nobody actually asked.

Promotion might allow product-trend dashboards after the classifier reaches the team’s coverage and low-risk error gates. Client billing can remain human-reviewed. Capitalization can remain discovery-only: the classifier retrieves candidate records, and finance makes the determination. One classifier does not need one universal trust level.

Test the privacy and evidence boundary

Semantic classification processes content, not only token counts. The classifier transcript can contain system instructions, user text, assistant text, tool names, calls, and results. Teams should therefore review what sensitive information reaches the chosen classifier model and provider.

OpenRouter’s data-collection documentation separates prompt storage, product-improvement opt-in, anonymous categorization, and metadata collection. Its Classifiers page says custom classification works even when input/output logging is disabled. Do not infer from that sentence that classifier processing is identical to not sending content. The feature still serializes content for the classification job.

Before activation, document:

• which model and provider perform classification;
• applicable provider logging and retention policy;
• whether private, regional, or zero-data-retention routing requirements carry over;
• which transcript fields may contain customer data or secrets;
• whether redaction occurs before classification;
• who can create or change classifiers;
• who can view semantic labels and related generation details;
• how deletion and incident response apply to classifier artifacts.

This note does not resolve OpenRouter’s contractual or regional behavior for a particular account. Confirm those details in the workspace configuration and current terms. If the team cannot establish an acceptable data path, use first-party metadata and local classification instead of treating observability as permission to broaden data access.

Know the failure modes before promotion

The most likely failures are not dramatic model hallucinations. They are quiet reporting errors:

Closed taxonomy pressure. The model chooses the nearest allowed value because no honest uncertainty route exists.

Truncation blindness. The evidence that changes the label appears after the per-turn limit.

Latest-turn confusion. A multi-turn workflow begins as research and ends as implementation, but the label definition never says which phase wins.

Coverage bias. Long or difficult requests fail classification more often, making the tagged subset look cheaper and simpler.

Sample bias. A request sample represents volume but not spend, risk, language, or time.

Taxonomy drift. A new team, product, or workflow changes the meaning of old categories while dashboards continue the same series.

Model drift. The selected classifier model changes behavior or availability.

Policy laundering. A probabilistic task label is passed into accounting, compliance, or client billing without a separate decision rule.

Double billing confusion. The cost of classification is omitted from the cost report it helped create.

Google’s work on data validation for machine learning highlights how systems can continue operating despite unexpected patterns and training-serving skew. A semantic spend system needs the equivalent: validate the inputs, output coverage, and cohort mix continuously, not only the model on a frozen test set.

Use a promotion receipt, not a green dashboard

The decision artifact should make missing evidence visible. A proposed receipt:

classifier_promotion:
  workspace: null
  decision_date: null
  owner: null
  allowed_use:
    exploratory_trends: false
    budget_planning: false
    client_billing: false
    finance_screening: false

  configuration:
    taxonomy_version: null
    classifier_prompt_version: null
    classifier_model: null
    sample_rate: null
    activation_window: null

  evidence:
    eligible_records: null
    tagged_records: null
    coverage: null
    high_consequence_coverage: null
    adjudicated_set_size: null
    per_label_precision_recall: null
    critical_error_count: null
    unknown_rate: null
    classifier_cost_share: null
    sampled_vs_population_check: null

  boundaries:
    metered_fields_separate: false
    unknown_bucket_enabled: false
    privacy_path_reviewed: false
    taxonomy_change_log: null
    human_review_required_for: []

  rollback:
    disable_owner: null
    disable_tested_at: null
    dashboard_annotation_plan: null
    next_review_date: null

  decision: hold

Approve one use at a time. exploratory_trends: true does not imply client_billing: true. Leave the receipt on hold when coverage cannot be reconciled, critical errors remain, the sample cannot support the claimed totals, or privacy routing is unresolved.

Rollback includes more than disabling the classifier. Annotate dashboards so readers know which periods used which taxonomy. Preserve prior version definitions. Reverse automated allocations made during an invalid window. Notify the owners of any downstream report that consumed the tags.

Where this protocol does and does not apply

This protocol is for small teams evaluating semantic labels attached to AI usage. It is not an independent benchmark of OpenRouter, the recommended model, or the six presets. A 120-record set can expose local ambiguity and obvious failure clusters; it cannot estimate rare errors with high confidence or prove performance across every language and workflow.

Human adjudication can be inconsistent or biased. Historical patterns can omit new work. Synthetic records can be cleaner than production traffic. Sampling tools may not expose every detail required for weighted estimates. The platform can change during beta. Precision and recall calculated on one taxonomy do not transfer automatically to another.

Do not use this method to automate tax, legal, employment, or regulated decisions. A classifier can retrieve candidate records or surface anomalies, but qualified owners retain the decision. Do not send production secrets into an experiment. Do not infer provider retention, regional processing, or contractual guarantees from general documentation.

The protocol is also unnecessary when declared metadata already answers the question. If each API key belongs to one cost center and shared work is rare, first-party ownership plus metered cost may be more accurate, cheaper, and easier to audit than semantic classification.

A 48-hour Build Lab plan

On day one, choose one low-consequence question, such as “Which work types drive AI usage?” Do not begin with client billing. Define four to six values plus unknown, write inclusion and exclusion rules, and freeze taxonomy version v1. Assemble 120 sanitized or synthetic records across common, mixed, long, tool-heavy, unknown, and high-consequence cohorts. Have two reviewers independently label a boundary subset, adjudicate disagreements, and revise the contract before exposing model output.

Configure one classifier model and prompt. Record the sampling rule and budget. Run the fixed ledger on demand where the product permits it. Calculate coverage first, then per-label precision and recall, critical errors, unknown handling, and classifier cost. Inspect the actual confusion pairs.

On day two, begin a shadow run. Reconcile eligible, sampled, tagged, failed, and unknown records against metered generation totals. Compare the sample with the wider population using model, cost, token, tool, language, and time fields. Review privacy routing and administrator access. Complete the promotion receipt for exploratory trends only.

At the end, choose one of four decisions: revise the taxonomy, change the classifier configuration, continue shadowing, or approve a narrow reporting use. If the team cannot explain the denominator, missing labels, error costs, and rollback, keep the tags out of budget truth.

OpenRouter Classifiers make the meaning of AI usage queryable. That can be a valuable observability layer. The Build Lab rule is to preserve the difference between a measured charge and a model’s interpretation of the work: calibrate the interpretation, price its mistakes, and promote it only for the decisions the evidence can support.

References

1. OpenRouter — Classifiers: Track What Your Agents Do and What It Costs
2. OpenRouter — Custom Classifiers documentation
3. OpenRouter — Usage Accounting
4. OpenRouter — Activity Export
5. OpenRouter — Data Collection
6. OpenRouter — State of AI report
7. Google for Developers — Classification metrics
8. Google Research — The ML Test Score
9. Google Research — Data Validation for Machine Learning
10. NIST — AI RMF Core, Measure
11. Bayerl and Paul — What Determines Inter-Coder Agreement in Manual Annotations?