LLM routing explained: picking the right model for each task

If you use several AI models by hand, you are already a router — you just run the scoring function in your head. "Code review? The code-tuned model. A quick lookup? Something fast and cheap. High stakes? The frontier model, and maybe a second opinion." An LLM router automates exactly that judgment, per request, with data instead of vibes.

What a router scores

A production router weighs four signals for every candidate model:

• Capability. Does the task need vision, tool calling, a huge context window, or deep reasoning? Models that lack a required capability are eliminated before scoring begins.
• Cost.Frontier models can cost an order of magnitude more per token than fast-tier models. Spending frontier compute on "reformat this list" wastes money the user could have spent on hard questions.
• Latency. A conversational turn should feel instant; a research task can afford to think. The router holds different latency budgets for different request types.
• Learned accuracy. The most interesting signal: how often has each model actually succeeded on this category of task, on this platform, recently? Static benchmarks age fast; live accuracy per task type is read from the database on every request, so the routing improves as usage accumulates.

Failover is part of routing

Providers have bad days: rate limits, elevated error rates, mid-stream disconnects, regional outages. A router that only picks the first model is half a router. The other half is the fallback chain — if the chosen model fails or degrades mid-answer, the request is retried on the next-best candidate automatically, ideally without the user noticing anything except that they still got an answer.

Why not just always use the best model?

Because "best" is per-task, not global. The model that tops reasoning benchmarks is rarely the fastest, is never the cheapest, and may trail a code-tuned competitor on refactoring work. Routing exists precisely because the frontier is jagged. A good router turns that jaggedness from a research finding into a product advantage: every subtask lands on the model that is best at it.

KernelFold's router does all of the above across a fleet of frontier models from multiple labs — you never pick a model, and you can audit which models were chosen and why in the live pipeline view. Developers can also bypass the router and call any fleet model directly through the API.