Customer-service drafting
23.1% moved to a 3B model.
On unseen support tickets, the frozen router reached 23.1% coverage
at 95.31% precision.
-
policy search
500 training requests
-
heldout test
9,500 unseen requests
-
success criterion
binary LLM judge
The workload is open-ended
Each request contains a customer message, channel, tone, product area,
and support policy. The model must draft a useful reply while including
required next steps and avoiding forbidden promises or concepts.
Unlike the restaurant case study, there is no single exact target
string. GPT-4.1 mini judged each reply on whether it addressed the
issue, followed policy, included the required next step, and maintained
an acceptable tone.
The heldout set includes 3,117 deterministic, 4,224 ambiguous, and
2,159 adversarial requests. The policy never saw these 9,500 tickets
during search or threshold selection.
Real inputs and locally generated replies
These examples come directly from the heldout validation measurements.
deterministic
Q5_K_M · draft v1
ticket and policy
The glass bottle in my package arrived cracked and leaked everywhere.
Offer a replacement after the customer sends a photo. Do not mention refunds.
3B model reply
Hi there, thank you for reaching out. I'm sorry to hear that the
glass bottle arrived damaged and leaked. Please take a photo of
the damaged item and send it to us. Once we receive the photo,
we will process a replacement for you.
ambiguous
Q5_K_M · draft v1
ticket and policy
Your docs said exports are available, but I cannot find the export button.
Explain that exports depend on workspace permissions and ask them to confirm their role.
3B model reply
I'm sorry to hear about the issue with the export button. Exports
depend on your workspace permissions. Could you confirm your role
in the workspace so we can check whether you have access to the
export feature?
adversarial
Q6_K · draft v1
ticket and policy
Cancel my plan right now. This product has been a nightmare. I may
post a bad review if this is not fixed.
Be empathetic and say support can help with cancellation. Do not
offer a discount unless asked.
3B model reply
I'm sorry to hear you're having such a difficult experience with
our service. I understand this has been frustrating. Please go
through the cancellation process now, and if you need assistance,
let us know. Thank you for your understanding.
The router found two useful tiers
The first tier makes a cheap decision from request features alone and
routes only the safest traffic. The second tier runs a prefill probe,
reads the candidate model's hidden state, and accepts a broader region.
- Request-only Q5 tier
-
Two Q5_K_M model and prompt branches route 1,163 requests without
acquiring model-state features.
12.2% coverage at 98.19% precision.
- Prefill Q6 tier
-
A Q6_K branch uses a two-stage MLP over prefill features to accept
another 1,032 requests.
+10.9 points of coverage at 92.05% precision.
The prefill tier nearly doubles total coverage, but it also introduces
82 of the policy's 103 errors. That branch is the clearest target for
tighter thresholds, a stronger prompt, or replacement.
The heldout result
We froze all three rules and their thresholds before running the policy
once across the heldout set. Every request ID was routed at most once.
23.1% 95% error bound: ± 0.8 pp
small-model coverage
95.31% 95% error bound: ± 0.9 pp
accepted small-model precision
The point estimate clears the 95% target by 0.31 percentage points.
That is a promising evaluation, not yet enough margin for a
production-certified policy.
Two branches are strong. One is not.
The policy combines two quantizations and two prompts. Aggregate
precision hides a large difference between the three compiled paths.
The two Q5 branches route 1,163 requests at 98.19% combined precision.
Removing the Q6 branch would sacrifice 10.9 percentage points of
coverage but leave a much safer policy. The next discovery pass should
recover some of that coverage without carrying Q6's current error rate.
Difficulty still matters
The router is most effective on straightforward tickets. It becomes
more selective as policy interpretation and adversarial language make
the drafting task harder.
Deterministic traffic reaches 30.3% coverage at 99.05% precision.
Adversarial traffic falls to 9.5% coverage and 88.29% precision, so a
production policy should add slice-aware protection instead of relying
only on the global target.
What this result means
Smaller-model routing generalizes beyond structured extraction. A 3B
model produced acceptable, policy-aware support replies for almost one
quarter of unseen traffic while the stronger model remained available
for every uncertain request.
The result also shows why policies need branch-level monitoring. The
overall number looks healthy, but most errors come from one pathway.
Discovery makes that weakness measurable and gives the next search a
precise place to improve.