17.1% moved to a smaller model.

The workload

The demo turns ordinary restaurant-search text into one JSON object: query, price levels, opening time, and maximum distance. A response only counts as correct when all four fields match the target.

The 1,000-request fixture deliberately mixes straightforward, ambiguous, and adversarial language. More than half the requests require some interpretation rather than direct field copying.

What the model actually sees

These are real fixture rows and their exact expected outputs.

{
  "maximum_distance_miles": 1.0,
  "open_after": "18:00",
  "price_levels": ["$$"],
  "query": "korean bbq"
}

{
  "maximum_distance_miles": 0.5,
  "open_after": null,
  "price_levels": ["$", "$$"],
  "query": "ethiopian food"
}

{
  "maximum_distance_miles": null,
  "open_after": "00:00",
  "price_levels": ["$", "$$"],
  "query": "dumplings"
}

A single smaller model is not enough

The 1.5B model is fast but only produces the exact JSON 52% of the time. The 3B model reaches 64%, but takes 76% longer on the hot path and still misses more than one request in three.

The useful question is not which model wins globally. It is which requests each cheaper model can handle reliably enough to avoid the stronger, more expensive fallback.

The router reads model state as generation unfolds

Routing is a sequence of checkpoints. The router starts with the raw request, then lets the candidate smaller model process the prompt. At prefill, before the first output token, it reads the model's hidden states and next-token logits. If more evidence is needed, it can inspect those signals again after the model generates 1, 2, 4, or 8 tokens.

1. checkpoint 1 request Text features only. The smaller model has not run yet.
2. checkpoint 2 prefill Read hidden states and logits before any output token.
3. checkpoints 3-6 1 / 2 / 4 / 8 tokens Read model state again while generation is underway.
4. final checkpoint full response Wait for the completed output and verify it.

How to read AUROC: it measures whether the routing signal ranks requests the smaller model will answer correctly above requests it will miss, across all possible routing thresholds. A score of 0.50 is chance and 1.00 is perfect separation. At 0.84, the prefill signal ranks the correct request as safer about 84% of the time when comparing one correct and one incorrect example.

Prefill raises routing AUROC from 0.70 to 0.84. Reading model state during the first eight generated tokens adds a smaller gain. A full verifier reaches 0.88, but takes more than three seconds, so most of the useful routing signal is available before the response is done.

Better routing signals unlock more coverage

The chart compares two different routing strategies. They differ in what evidence the router can inspect, when it makes the decision, and how many smaller-model pathways it can choose from.

Strict request-only gate

Decides before the smaller model runs, using features such as request length and whether time, price, distance, or ambiguity markers appear. Its acceptance threshold is extremely conservative. Malformed or schema-invalid outputs use the stronger fallback. 2.7% coverage at 97.4% precision.

Frozen prefill router

Processes the prompt first, then uses hidden-state and logit features in four learned routing rules. Those rules can choose between two smaller model/prompt configurations or the stronger fallback. "Frozen" means the rules and thresholds were fixed before the 9,500-request heldout test. 17.1% coverage at 95.6% precision.

These points come from separate validation runs, so this is a comparison of routing strategies rather than a controlled sweep of one threshold. The prefill router achieved more than six times the small-model coverage at 1.8 percentage points lower precision.