07 - Model Selection and Routing¶

A production AI system should not assume that every request belongs to the strongest available model. Different tasks have different requirements for quality, latency, cost, context size, multimodality, structured output, and tool use.

Model selection is therefore an architectural decision, not just a configuration field.

Why routing matters¶

Imagine an application that performs:

language detection,
document classification,
support-answer generation,
difficult debugging,
image understanding,
embeddings.

Sending every task to one expensive reasoning model may work, but it is often wasteful and slower than necessary.

A better mental model is:

Request
   |
Task classification
   |
   +--> simple extraction ------> small/fast model
   +--> complex reasoning ------> strong reasoning model
   +--> image task -------------> multimodal model
   +--> semantic indexing ------> embedding model

Main selection dimensions¶

Capability¶

Can the model reliably solve the task?

Examples:

complex code reasoning,
long-form synthesis,
extraction,
classification,
image understanding,
tool calling.

Do not optimize price before establishing a minimum acceptable quality level.

Latency¶

A model that is excellent but takes 20 seconds may be unsuitable for autocomplete or interactive UI.

Different workloads tolerate different latency:

Autocomplete        -> very low latency
Chat assistant      -> moderate latency
Background analysis -> higher latency acceptable

Cost¶

Cost is affected by more than model price:

input tokens,
output tokens,
reasoning effort,
request volume,
retries,
context size.

A model that is twice as expensive per token may still be cheaper if it solves the task in one call instead of three failed attempts.

Context size¶

Some tasks require large documents or repository context. Others need only a few hundred tokens.

A large context window is useful only when the task actually requires it.

Structured output and tool support¶

If the application relies on schema-constrained output or tool calls, support quality for those capabilities matters as much as natural-language quality.

Multimodality¶

Tasks involving images, audio, screenshots, or documents may require a model that accepts those modalities directly.

Static model selection¶

The simplest strategy is to assign a fixed model to each application feature.

invoice extraction -> Model A
support chat       -> Model B
code analysis      -> Model C

This is easy to understand and debug and is often a good starting point.

Dynamic routing¶

A more advanced system can choose a model per request.

Example:

User request
   |
Router
   |
   +--> simple FAQ ----------> fast model
   +--> account investigation -> stronger model + tools
   +--> difficult debugging --> reasoning model

Routing inputs may include:

task type,
input length,
user tier,
required latency,
confidence of a previous model,
risk level.

Escalation pattern¶

A useful approach is cheap-first with escalation.

Fast model
   |
Can solve confidently?
   |
  yes -> return
  no  -> stronger model

But this only works if escalation can be detected reliably. Asking a model whether it is confident is usually not enough by itself. Use measurable signals where possible:

schema failure,
missing required evidence,
deterministic validation failure,
low retrieval quality,
evaluation-backed classifier.

Fallbacks¶

Model routing also improves resilience.

Primary model unavailable
        |
Fallback compatible model

A fallback must be tested. Two models may differ in:

tool-call format,
output quality,
context limits,
instruction following,
latency.

Do not assume drop-in compatibility without evaluation.

Example: support platform¶

Possible architecture:

Incoming request
      |
Intent classifier
      |
      +--> simple FAQ -> small model + retrieval
      |
      +--> refund request -> strong model + account tools
      |
      +--> legal escalation -> deterministic workflow + human

Notice that some routes should not lead to an LLM at all.

When not to route¶

Routing adds complexity:

more configuration,
more tests,
model-specific behavior,
harder observability,
more failure modes.

For a small system, one well-chosen model may be better than premature routing infrastructure.

Observability¶

Record at least:

selected model,
reason for selection,
token usage,
latency,
retries,
validation failures,
outcome/evaluation score where available.

Otherwise routing optimization becomes guesswork.

Developer takeaway¶

Choose models according to the workload, not prestige.

Start simple, measure quality/latency/cost, and introduce routing when there is evidence that different task classes benefit from different models.