Use Smaller Models.

Pay less. Respond faster. Run on-device for effectively zero recurring inference cost.

Most applications send every request to the same large cloud model. That is simple, but expensive. Many requests do not need the strongest model in the stack.

A smaller model can often produce the same acceptable result for a fraction of the cost and latency. Run it through our cloud for a cheaper managed path, or run it on the user's device and reduce the recurring inference cost for that traffic to effectively zero.

The hard part is knowing which model is good enough for each request. Should it go to a quantized 1.5B model on-device, a 3B model through our cloud, or the larger model you use today? The cheapest acceptable answer depends on the request, available hardware, model quality, latency, memory, power, and the shape of the user session.

SmallerModels searches across those tradeoffs and learns an inference policy for your workload. For each request, it selects the smallest, cheapest pathway expected to satisfy your application's quality and performance requirements.

In production, the router applies that policy across on-device and hosted models, escalating only when a cheaper model is unlikely to succeed. The application keeps one behavioral contract while its inference cost and latency fall.

As traffic changes and new models, quantizations, hardware, and pricing become available, SmallerModels can search again. The goal is to keep every request on the lowest-cost path that still works, and to move as much eligible traffic on-device as possible so it stops generating a recurring inference bill.

Discover, Route, Execute

Discover

An offline inference-policy search engine. It observes representative application traffic and continuously searches across models, quantizations, prompts, execution locations, and hardware classes to determine the cheapest model pathway that can handle each operation and when it should fall back to a stronger model.

We model discovery as a constrained contextual bandit because each request arrives with observable context \(x\), each feasible inference pathway is an action \(a \in \mathcal{A}\), and the system must learn which action should handle the request while respecting production constraints.

The key quantity is the conditional success model:

\[ \hat{q}(x,a) \approx P(\text{answer is acceptable} \mid x,a) \]

The optimizer is not simply asking which model is best. It is solving a constrained assignment problem:

\[ \pi^* = \arg\min_{\pi \in \Pi} \mathbb{E}_{x \sim \mathcal{D}} \left[C(x,\pi(x),s)\right] \]

subject to:

\[ P(\text{acceptable} \mid \pi(x)) \ge q_{\min} \]

\[ L(x,\pi(x),s) \le L_{\max}, \qquad M(\pi(x),s) \le M_{\max} \]

\(C\) is inference cost, \(L\) is latency, \(M\) is memory, \(s\) is execution state, and \(q_{\min}\) is the required quality threshold. For an on-device pathway, the recurring component of \(C\) can be effectively zero.

What makes this different from a textbook bandit is that actions are compositional. A pathway includes model family, quantization, prompt, execution location, context size, decoding policy, model residency, prefix/cache state, checkpoint strategy, and fallback behavior. Some knobs change answer quality; others change cost, feasibility, latency, memory, or future runtime state. Discovery has to reason over all of them.

In the larger system, discovery is the offline evidence-acquisition and policy-search stage. It consumes captured traffic, evaluator evaluator outcomes, synthetic expansions, model/runtime measurements, and device constraints. It selectively measures request-path pairs, estimates where smaller pathways succeed, compares cheaper challengers against stronger incumbents, and searches for the minimum-cost policy under quality and systems constraints.

Its output is a frozen candidate policy: small-model serving regions, the inference paths assigned to those regions, uncertainty estimates, expected coverage, fallback behavior, and the runtime assumptions needed to execute safely.

A discovered router policy can look like this. The score checks are cheap router gates; a candidate smaller model is only invoked after a rule accepts.

{
  "serving_policy": "lowest_cost_acceptable_pathway",
  "quality_target": 0.95,
  "rules": [
    {
      "when": "score(qwen1.5b-q4km) >= 0.795",
      "then": "run qwen1.5b-q4km on_device",
      "runtime": {
        "context_tokens": 512,
        "residency": "HOT",
        "shared_prefix_state": "shared_prefix_hot",
        "estimated_latency_ms": 30,
        "estimated_memory_gb": 1.50
      },
      "recurring_inference_cost": "effectively_zero",
      "fallback": "next_rule"
    },
    {
      "when": "score(qwen3b-q4km) >= 0.905",
      "then": "run qwen3b-q4km in_smaller_models_cloud",
      "runtime": {
        "context_tokens": 512,
        "residency": "HOT",
        "shared_prefix_state": "shared_prefix_hot",
        "estimated_latency_ms": 234,
        "estimated_memory_gb": 2.55
      },
      "fallback": "strong_model"
    }
  ],
  "fallback": "existing_strong_model"
}

Discovery carves request traffic into conditional serving regions.

Route

A production policy layer. For each request, it considers the operation, device state, session context, and application requirements, then chooses the cheapest model and execution location expected to work.

Router compilation turns the discovered offline policy into an compact executable decision program. Discovery may find that a region \(R\) can be handled by pathway \(a\), but production needs a legal recognizer \(g(x,z)\) that can decide membership at request time using only available signals.

Those signals can include request features, runtime state, and optionally model-state features \(z\) acquired during a lightweight prefill or probe pass.

The compiler objective is closer to:

\[ g^*, a^* = \arg\max_{g,a} P(g(x,z)=1) \]

subject to:

\[ P(\text{acceptable} \mid g(x,z)=1,\ a) \ge q_{\min} \]

\[ C_{\text{signal}}(z) + C_{\text{inference}}(a) \le C_{\text{budget}} \]

So the compiler is not just training a classifier. It is lowering a discovered policy into an efficient cascade: grouping compatible rules, sharing signal-acquisition passes, calibrating conservative thresholds, and emitting MLP gates that can run on-device or beside a hosted inference service.

Sample two-stage MLP router.

stage 1: cheap MLP gate

Reject requests that are clearly not worth probing.

Inputs: operation, text, device state, session context
Output: probe the smaller model or use the strong fallback

stage 2: hidden-state MLP gate

Use the model's own representation to decide smaller model versus stronger fallback.

Inputs: pooled activations, logits, confidence signals
Output: smaller model when confident, stronger model when uncertain

In production, multiple logical rules should not necessarily mean multiple model passes. If several rules depend on the same model, prompt, or prefix, the compiler should acquire the shared probe once, extract all required features, and evaluate the gates in memory.

The key innovation is using hidden-state information as router training signal. The router is trying to predict the smaller model's competence boundary, and that boundary is partly encoded inside the model's own representation of the request.

Surface text features can say what the request looks like to us; hidden states say what the request looks like to the model. By training MLP gates over these model-state features, the router learns a boundary in the model's latent geometry instead of relying only on brittle lexical features. This is why prefill-derived signals can improve segmentation: the smaller model helps expose the shape of its own reliable region before generation.

Examples of hidden-state features we consider include:

Pooled activations from selected prefill layers, which capture how the model internally represents the request before it generates.
Projected activation vectors, which compress that representation into a small feature space the router can evaluate cheaply.
Request-token window features, which help the router focus on the actual user request instead of shared prompt boilerplate.
Full-prompt prefix features, which can reveal whether the prompt, schema, or task framing puts the request in a familiar region.
Next-token logits summaries, such as entropy or top-token margin, which indicate how confident the model appears before answering.

Hidden-state features make the smaller model's competence region easier to compile.

raw traffic

hidden-state embedding

compiled router

Execute

The inference runtime. It carries out the policy on-device or through our cloud, managing model residency, memory, scheduling, and execution across supported hardware. When a pathway runs on-device, inference is paid for by hardware the user already owns instead of a recurring API call.

The cheapest model is a moving target.

Which operations can move to a smaller model without changing product behavior?
Which model, quantization, prompt, and execution location should each operation use?
Which pathways can run on-device and eliminate recurring inference cost?
When should a model stay resident, and when should memory be released?
How should routing react to price, latency, battery, memory pressure, device class, and session shape?
When should a previously good policy be rediscovered?