Updated October 12, 2026

What we measured so far

Every version of Switchboard is tested on the same 10,000 prompts it never saw during training. We publish the numbers that look bad too, and you can download every answer below.

Routing quality

10,000 held-out prompts in their natural mix: 6,931 easy, 1,716 medium and 1,353 that need a frontier model. Full-precision models, greedy decoding.

Cost saved

Higher is better

How much cheaper the bill is than sending every prompt to Claude Opus 5.5.

Switchboard v0.164.5%

Switchboard v067.8%

Perfect router76.2%

Always small97.5%

Prompts sent too low

Lower is better

Prompts that needed the frontier model but went to a cheaper one. This is the mistake you'd notice.

Switchboard v0.15.5%

Switchboard v07.1%

Random router, same budget10.0%

Always small13.5%

Hard prompts caught

Higher is better

Of the prompts that needed the frontier model, the share Switchboard sent there.

Switchboard v0.159.1%

Switchboard v047.5%

Random router, same budget26.2%

Ranking hard above easy (AUC)

Higher is better

How well the model orders prompts by difficulty, whatever the cut-off. 0.5 is a coin flip, 1.0 is perfect.

Switchboard v0.10.787

Switchboard v00.785

Coin flip0.500

Unreadable answers

Lower is better

Answers that weren't valid JSON, out of 10,000. Those would fall back to frontier.

Switchboard v0.10

Switchboard v00

The random router sends the same share of prompts to frontier as Switchboard v0.1, picked at random. A perfect router knows every answer in advance.

Does every build route the same way?

Smaller files make the model faster to download, but they can change its decisions. We checked each build against the full model on 347 test prompts: 300 picked at random from the 10,000 above, plus 47 borderline ones that cover all three tiers.

Build	Size	Same decision
GGUF F16Ollama 0.40.3 via ollama run hf.co/..., temperature 0 from the repo params347 in order, 346 when the prompts were sent in random order (Ollama reuses cached prompt prefixes)	994 MB	347 of 347100.0%	Shipped
Browser ONNX, 8-bitTransformers.js 4.2.0, WebGPU, desktop Chrome299 of 300 random, 45 of 47 borderline; domain 347 of 347	660 MB	344 of 34799.1%	Shipped
GGUF Q8_0Ollama 0.40.3, GPU338 of 347 on CPU	531 MB	333 of 34796.0%	Not shipped
GGUF Q4_K_MOllama 0.40.3, GPU	398 MB	325 of 34793.7%	Not shipped
GGUF Q6_KOllama 0.40.3, GPU	506 MB	320 of 34792.2%	Not shipped
Browser ONNX, 4-bit layersonnxruntime 1.24, CPUmeasured on the 300 random prompts only	463 MB	197 of 30065.7%	Not shipped

For a router, a changed decision is a wrong answer, so we only ship builds that match the full model. 4-bit builds of a 0.5B model lose too much precision on the borderline prompts.

Speed

Time to get one routing decision, measured on one desktop PC.

Setup	Median	Slowest 10%
GGUF F16 in Ollama, desktop CPU	281 ms	336 ms
Browser ONNX, WebGPU in desktop Chromeafter the one-time 660 MB download	400 ms	460 ms

Every answer, downloadable

One JSON line per prompt: the prompt, its label, the Mixtral score behind the label, Switchboard's raw output and decision, and whether the prompt was served well.

Raw outputs v0.1 6.0 MB, 10,000 lines
Raw outputs v0 6.0 MB, 10,000 lines

How we tested

Prompts: The validation split of routellm/gpt4_dataset (Apache-2.0, from LMSYS): 10,000 real prompts from chat logs, instruction sets and coding questions. None were used in training.
What "needs frontier" means: Every prompt comes with a 2023 Mixtral-8x7B answer that a GPT-4 judge scored from 1 to 5. A score of 5 means a small model is enough, 4 means mid, and 1 to 3 means frontier. A 2026 small model is stronger than Mixtral-8x7B, so this label is cautious: some prompts marked frontier would be fine on a smaller model.
Served well: A prompt counts as served well if Switchboard sent it to frontier, or sent it lower and the Mixtral answer scored 4 or 5.
Cost: Output price per 1M tokens on Surplus: Claude Haiku 5.5 ($0.50) for small, Claude Sonnet 5.5 ($10) for mid, Claude Opus 5.5 ($20) for frontier. Savings compare the routed bill with sending every prompt to Opus.
Settings: Greedy decoding (temperature 0), at most 24 new tokens, prompts cut to 512 tokens, the default chat template with no extra instructions.
Builds: The full model in fp32 is the reference. Each build answered the same prompts and we counted how often its tier matched. GGUF builds ran in Ollama 0.40.3, the browser build in Transformers.js 4.2.0 with WebGPU in desktop Chrome.
Where it ran: One desktop PC with an RTX 5070 Ti, for training, evaluation and the speed numbers.
Limits: The labels come from models and a judge, not from people. Most prompts are English. Each number is one run, so differences of a few tenths of a point don't mean much. On prompts that sit right between two tiers, tiny numeric differences can flip the decision.