Skip to main content
Smart Model Routing puts a decision engine in front of every governed model call. For each request — and each step an agent takes inside a run — Egis asks one question: is the model this code asked for actually the right model for this request? Three answers are possible:
  • Downgrade — the request is simple enough that a cheaper model in the same family (or another provider, on the Gateway with vault keys) will produce an equivalent answer. The call is served on the cheaper model and the estimated saved dollars are audited per call. The dashboard labels these calls Est. cost-optimized.
  • Upgrade — the request needs more capability than the requested model offers (deep reasoning, complex code, very long context). The call is served on a stronger model; the extra spend is audited as a deliberate quality investment. The dashboard labels these calls Est. quality-optimized.
  • Keep — the requested model is the right fit. Nothing changes, nothing is logged as routed.
Routing never touches what your code receives. The response arrives in exactly the shape your framework expects — same client library types, same fields — with the served model on the response’s model field and an additive egis.routing marker for programmatic consumers.

​
How a decision is made

The decision engine lives on the Egis platform (not in the SDK), so it can be tuned continuously without an SDK release:
  1. Classify. A small, fast LLM classifier reads the request’s post-sanitization, label-redacted preview — raw prompt text never rides on the decision call — and scores its complexity and needs (reasoning, code, long context). A deterministic heuristic backs the classifier when the LLM is unavailable.
  2. Select. A deterministic selector matches that signal against the model catalog: each routable model carries a capability tier and current per-token prices. The selector only ever picks models that your allow_model policies permit and that the calling process can actually authenticate against.
  3. Apply. The SDK (or the Gateway) swaps the model before the forward. Same-provider swaps re-use your existing client and auth unchanged. Cross-provider swaps translate the payload, execute against the target provider, and translate the response back — and only happen when a faithful translation is guaranteed (plain-text, non-streaming, tool-free calls).
Decisions are cached aggressively in-process, so an agent loop re-sending near-identical prompts pays for one decision, not one per call.

​
Framework coverage

  • OpenAI (Chat Completions and the Responses API) and Anthropic (messages.create) — same-provider swaps on every call; cross-provider for plain-text, non-streaming, tool-free chat when the process holds the target provider’s key.
  • google.genai (Gemini, sync/async, streaming included) — same-provider swaps on your own configured client.
  • Claude Agent SDK — a fresh same-provider decision per turn. The module-level query() routes before the subprocess boots; persistent ClaudeSDKClient conversations are switched on the live session (via the SDK’s set_model control request) right before each turn’s prompt ships, and restored to the configured model when the engine stops routing.
  • Frameworks built on these clients (LangChain, LlamaIndex, openai-agents, CrewAI, …) inherit routing through the patched underlying client automatically.

​
Fail-open, always

A routing swap must never make a call fail that would otherwise have succeeded:
  • The decision service unreachable, slow, or erroring → the call runs on the requested model.
  • A routed call itself fails (provider error, revoked key) → the SDK retries once on the originally requested model before surfacing any error.
  • A payload too rich to translate faithfully (images, tool results) → cross-provider routing is skipped for that call.

​
Where the results show up

  • Model Center — the master switch plus live stats: routed calls, the downgrade/upgrade split, estimated dollars saved, estimated dollars invested in quality, an estimated savings timeline, and the top requested→served pairs.
  • Requests page — every routed call’s detail modal shows the requested model, the served model, the decision’s reason, and the estimated signed cost delta computed from the real token usage.
  • Run timeline — each routed step inside an agent run carries a chip (“Est. cost-optimized from claude-opus-4-8”) so multi-step routing is auditable step by step.
Every dollar figure is an estimate, and the dashboard says so everywhere it renders one. Savings are recomputed server-side from the served call’s real token counts priced against the requested model’s list price — the SDK never self-reports a dollar figure — but the requested model never actually ran, so the comparison is a counterfactual, not a billed amount.

​
Turning it on and off

Routing is a plan-tier feature (Growth and Enterprise) and ships enabled for entitled orgs:
  • Org-wide — the master switch on the Model Center. Flipping it reaches every connected SDK within ~50 ms over the live event stream; no restart needed.
  • Per agent — the routing switch on the agent’s identity modal pins routing on or off for that agent regardless of the org switch.
When routing is off (either level), the SDK goes fully dormant for the affected calls: zero decision requests, zero added latency.

​
Latency budget

The decision round-trip is bounded by a 3-second timeout and is almost always a cache hit after an agent’s first call. The time spent deciding is kept out of the call’s policy latency column — that number tracks what evaluating your policies cost, so turning every policy off makes it read zero whether or not routing is running. On the inline Gateway the routing window is reported on the X-Egis-Timing response header instead:
gate is everything Egis did before forwarding upstream, policy is evaluation only, and routing is the decision round-trip.