How I built a private LLM router: one boring lookup table decides what leaves my network

The first version of my LLM router worked out the privacy decision for you. If a request arrived without a job label, the router read the prompt text and guessed the category from keywords. That is how it leaked.

I caught it while reading the request journal one afternoon. Some prompts about internal documents had matched the word “research” in their own text, been auto-labelled as research, and gone to Gemini. The leak itself was small, but nothing bounded it: once you trust a keyword-matcher with your privacy policy, you have trusted it with every prompt it has not seen yet. The same router also had a rule that if a prompt was too big for the local model’s memory, skip the local machine and use the cloud, which reads as sensible engineering right up until the oversized prompt is also a sensitive one.

Both behaviours got deleted in a single patch the next day. What replaced them is the only original idea in this project, and it is boring enough that you can lift it into your own stack in an afternoon: the task label’s prefix is the privacy boundary. A label beginning local_ never leaves my network. A label beginning cloud_ may. No keyword matching, no classifier, no inference about what the prompt contains. One lookup table, sixteen lines, and an unrecognized label returns an HTTP 400 rather than a guess.

This is the sequel to the manual routing playbook I published earlier. That post was about picking the model by hand for each job. This is the machine that does the picking, and the one idea in it worth stealing.

Should you build this? Probably not. Use LiteLLM.

Everyone competent asks this first, so it goes second, and it deserves a straight answer rather than a straw man.

OpenRouter is a hosted reseller of cloud models with real privacy machinery: you can refuse providers that train on your prompts, filter a request by the provider’s data policy, and buy in-region routing on the enterprise tier (privacy and logging docs). What it cannot do is send a request to a graphics card in my basement. That is not a gap in OpenRouter, it is outside what OpenRouter is.

LiteLLM is a different matter, and it is genuinely good software. You self-host it, it normalizes about a hundred providers behind one OpenAI-compatible endpoint, and it ships load balancing, budgets, health checks, retries, and fallback chains. It also does tag-based routing: you tag each deployment in the config, the caller passes tags in the request body or an x-litellm-tags header, and the router only considers deployments carrying that tag (tag routing docs). You can approximate everything below with it. The honest argument is therefore not a capability gap, it is about how strong the guarantee is and how fast you can audit it, and it comes down to two defaults.

Tag routing fails open. An untagged request goes to whatever deployments carry the default tag, and if nothing carries that tag, every healthy deployment is eligible. The docs put it plainly: default-tagged deployments if any exist, otherwise all deployments. That is correct for a load balancer, whose job is to find the request a home. It is wrong for a trust boundary, where the request you have to worry about is the one nobody remembered to tag. My router refuses that request. LiteLLM, out of the box, finds it a machine.

Fallbacks are the mechanism that crosses the boundary. A fallback chain exists to move a request elsewhere when the first choice is busy or broken, which is precisely the moment a private prompt escapes. LiteLLM can disable them, per request or per key (reliability docs). You have to remember to, on every key, forever. In my table the local half has no cloud entry to fall back to, so there is nothing to remember.

A safe LiteLLM configuration exists. It is one you have to get right and keep right across tags, fallback chains, default fallbacks, and per-key metadata, which is four surfaces on which a future version of me can be careless. When somebody asks where a prompt went, I want to answer by reading one dictionary, not by reasoning about precedence between four settings. That is the entire argument, and if it does not describe your problem, use LiteLLM and take the pattern with you.

Privacy-by-prefix: four rules, no cleverness

Every request carries a task_type, a plain string saying what the work is: classify, write, reason, generate code. The prompt itself is never inspected. Four rules do the rest.

• The label is required. Missing or unrecognized returns HTTP 400, before the prompt is read and before a backend is chosen.
• The prefix decides the tier, and it decides first. local_ goes to my own hardware, cloud_ may go to an outside provider. Load, quota, and model quality are within-tier questions that never get to reopen the boundary.
• The local tier never spills. If no local machine has room, the request fails. There is no cloud target on that side of the table for a busy request to find. An error beats a prompt on someone else’s server.
• The declaration lands in the log. A journal line reading task_type: cloud_classifyextract records that somebody declared this prompt cloud-safe at that moment. That is what made the original leak findable.

The caller decides by choosing the label, which is the point. A separate privacy: true field is a thing you can forget, and whatever default it carries is the leak waiting to happen. A prefix is visible at the call site, in the log line, in the error, and in code review, and it costs six characters. Underneath, my machines sit on one Tailscale mesh, a private tunnel connecting my own hardware as if it shared an office LAN, and a local_ request rides that tunnel without touching the public internet.

The whole trust decision is one lookup table

Here is the table the router runs today. Where a lane lists more than one target, the request goes to whichever is least busy, and the others stand behind it as fallbacks inside that tier.

task_type (the label)	Eligible backends, least-busy wins
cloud_research	Gemini gemini-2.5-flash (web grounding)
cloud_reasoning	Cerebras gpt-oss-120b, then Groq gpt-oss-120b
cloud_code	Codex gpt-5.4, then Codestral
cloud_writing	Mistral mistral-small-2603
cloud_classifyextract	Groq llama-3.1-8b-instant, then Cerebras gpt-oss-120b
cloud_other	Escape hatch: the caller names the model
local_classifyextract	Ministral-8B on the Quadro sidecar, Ministral-8B on the M4000, or the 3090 Ti (qwen3.6-35b-a3b)
local_code	Mac Studio (qwen3.6-27b-mtp), or the 3090 Ti (qwen3.6-35b-a3b)
local_reasoning	3090 Ti (qwen3.6-35b-a3b)
local_reasoning_nothink	3090 Ti, thinking forced off, or the Mac Studio (qwen3.6-27b)
local_reasoning_secondary	Mac Studio (qwen3.6-27b-mtp)
local_research	3090 Ti
local_writing	3090 Ti (qwen3.6-35b-a3b)
local_synthesize	Mac Studio (qwen3.6-27b-mtp), over retrieved context
local_image	Mac Studio (qwen3-vl-4b-instruct)
local_promptinjecttest	Mac Studio (hermes-3-llama-3.2-3b)

One bit of vocabulary from that table. A mixture-of-experts model, or MoE, keeps a large pool of weights on hand but only runs a small slice of them for each token it produces. That slice is the model’s active parameters. The 3090 Ti’s qwen3.6-35b-a3b holds 35 billion parameters and activates about 3 billion per token, which is why it answers at closer to the speed of a much smaller model.

In code, the trust decision is a dictionary mapping each label to the tier it belongs to. No precedence rules, no pattern matching, nothing to reason about.

_ROUTES: dict[str, Tier] = {
    # cloud-only categories
    "cloud_research":           Tier.GEMINI,    # web grounding
    "cloud_reasoning":          Tier.CEREBRAS,  # gpt-oss-120b
    "cloud_code":               Tier.CODEX,     # gpt-5.4 (ChatGPT auth = effectively free)
    "cloud_writing":            Tier.MISTRAL,   # Gemini Pro quota is zero on current API keys
    "cloud_classifyextract":    Tier.GROQ,      # llama-3.1-8b-instant
    "cloud_other":              Tier.CLOUD_ANY, # caller names the model

    # local categories
    "local_classifyextract":    Tier.LOCAL,     # ministral-8b sidecars, 3090 Ti behind them
    "local_code":               Tier.LOCAL,     # Mac Studio qwen3.6-27b-mtp, 3090 Ti behind it
    "local_reasoning":          Tier.LOCAL,     # 3090 Ti
    "local_reasoning_nothink":  Tier.LOCAL,     # same, thinking forced off
    "local_reasoning_secondary":Tier.LOCAL,     # Mac Studio qwen3.6-27b-mtp
    "local_research":           Tier.LOCAL,     # 3090 Ti, caller pre-fetches context
    "local_writing":            Tier.LOCAL,     # 3090 Ti
    "local_synthesize":         Tier.LOCAL,     # Mac Studio, RAG over retrieved context
    "local_image":              Tier.LOCAL,     # Mac Studio qwen3-vl-4b-instruct
    "local_promptinjecttest":   Tier.LOCAL,     # Mac Studio hermes-3-llama-3.2-3b
}

The request flow has the same shape: validate the label, route by prefix, pick a machine on that side of the line, adjust the prompt if the backend needs it, dispatch, log.

  Request arrives at /v1/chat/completions
          |
          v
  ┌─ task_type in body? ── no ──> HTTP 400 (explicit required)
  │         yes
  │         v
  │  task_type valid? ── no ──> HTTP 400
  │         yes
  │         v
  │  route by prefix:
  │    local_*  -> Tailscale-only backends (Mac Studio / 3090 Ti PC)
  │    cloud_*  -> external (Groq / Cerebras / Codex / Gemini / Mistral)
  │         |
  │         v
  │  within-tier selection:
  │    least-loaded backend that can serve this task
  │    ties broken round-robin
  │    pinned tasks (writing, image, judge) go to their one machine
  │         |
  │         v
  │  prompt adaptation:
  │    local+classifyextract:  disable thinking (saves ~2950 tokens)
  │    local:                  cap max_tokens, lower temp
  │    cloud_research:         enable Google Search grounding
  │         |
  │         v
  └──> dispatch, journal, update quotas

To put this in your own stack you need three things, none of them my code. A validated set of labels, so an unknown one is rejected rather than defaulted. A prefix check that runs before backend selection, so nothing downstream can reopen the question. And no edge in your fallback graph leading from the private half to the public half, so a busy local tier produces an error instead of an escape. Everything below that line, which backend, which model, which quota, is your infrastructure and none of my business.

Why a declared label and not a cascade or a classifier

The obvious alternative is a cascade: try the cheapest model first, climb the price ladder only when it fails. My local Qwen scores 40% on debugging tasks (full numbers in the 38-task benchmark post), so sending every debugging prompt to Qwen first buys a wasted round trip and a garbage answer before the model that can actually do the work ever sees it. That benchmark drove the whole design: free local models hold their own on classification, extraction, code generation, and writing, and they lose meaningfully on multi-step reasoning and debugging. Which kind of work this is happens to be something the caller already knows, before the request is sent rather than after it fails.

The fashionable alternative is semantic routing: a small classifier model reads the prompt and picks a backend from what the prompt looks like it is doing. That is what version one of my router did, in its cheapest possible form, and it is the thing that leaked. Swapping the keyword matcher for a small model makes the guess better. It does not make the guess accountable. At a trust boundary I do not want better odds, I want no odds.

What will bite you

Silent failures stack. A routine job started returning 502s on every classify request, and the journal showed empty responses coming back from the 3090 Ti. The real bug was three layers down. The local backend had been prepending a text instruction (/no_think) to tell the model to skip its reasoning step, assuming the newer Qwen obeyed the same directive the older one did. It did not. The model ran its full hidden reasoning anyway, spent its entire output budget thinking, and returned an empty answer with nothing left to say out loud. Three silent choices, one for the tier, one for the machine, one for the model, produced three silent failures stacked on each other, and the 502 on top said nothing about any of them. Pinning each choice explicitly is what turned the quiet failures into loud ones.

Model swapping stalls the queue. The Mac Studio runs LM Studio with a text model and a vision model defined, and to save memory it loads only one at a time, swapping on demand. That swap takes 10 to 30 seconds, and every request waiting on that machine stalls for the whole window, so anything arriving mid-swap went from under a second to 30-plus seconds. Pinning a default text model fixed it: image traffic now pays the swap cost only when an image actually arrives.

Round-robin is the wrong default. The first version spread local work by taking strict turns, which works badly the moment your machines are not identical, because taking turns hands the next request to a box already three deep in a queue while the other sits idle. The router now picks the eligible backend with the smallest load number (requests in flight plus requests queued), and round-robin survives only as the tie-break.

Six weeks later, the blast radius stayed inside one table

The stack underneath this router has churned harder in the last six weeks than in any six weeks before it, and every one of those changes landed as an edit to a lookup table.

Cerebras deleted a model out from under me. The classify lane ran on Cerebras llama-3.1-8b-instant, and Cerebras pruned that model from my account entirely. I found out while researching the free LLM API teardown, and would otherwise have found out as a wall of failures in a batch job. The repair was one line: Groq took over as primary for cloud_classifyextract, with Cerebras gpt-oss-120b behind it. No caller changed.

Three Gemini accounts died. Of six Gemini lanes, the three login-based ones have all gone unhealthy, and the health check that runs every 60 seconds pulled them out of rotation without my involvement.

Classify and extract merged into one lane. They had always routed to the same model with the same settings, so cloud_classify and cloud_extract became cloud_classifyextract. This is the only change in six weeks that broke callers, and it broke them the right way: every stale call site returned HTTP 400 with a list of the valid labels, loudly, on the first request.

The daily driver got bigger and faster at once. When I built the router, the 3090 Ti ran a dense qwen3.6-27b at about 42 tokens per second. It now runs qwen3.6-35b-a3b, the mixture-of-experts model from the table above, which I measure at about 136 tok/s end to end. The dense 27b moved to the Mac Studio and serves the code and synthesis lanes from there. Both models changed machines underneath the router, and the router did not care.

None of this was in the plan, and six weeks of churn is the honest argument for the pattern. I picked several of the models wrong, a provider deleted one out from under me, three accounts died, and my load-spreading strategy turned out to be the wrong one. The blast radius of all of it stayed inside one file I can read in twenty seconds, and no prompt crossed the line while any of it was happening.

What it costs

The cloud tier is net zero dollars. Groq handles the quick sorting-and-labelling work on a free quota I have never exhausted, Cerebras runs the cloud reasoning lane on another, and Codex through my ChatGPT login gives me a top-tier coding model with no metered bill. Work that genuinely needs Anthropic’s Claude still goes to Anthropic, and that is a real bill. I pulled the free tiers apart lane by lane in the free LLM API teardown, so here I will only give the caveats.

Free is doing some work in that sentence. Gemini is the thinnest lane at 5 requests per minute and 100 per day, enough for research questions and nothing like enough for bulk work. Codex launches as a separate process rather than a streaming connection, so it never sees the quick jobs, and using a ChatGPT login this way for automated pipelines sits in a grey zone of OpenAI’s terms, which were written for interactive use. The local tier is only free because the hardware is already paid for and the electricity is somebody’s rounding error, which is to say mine. It is not the slow tier I had budgeted for, either: at about 136 tok/s the 3090 Ti feels comparable to the cloud on most routes, and the latency tax is 1 to 3 seconds rather than the 10-plus I had penciled in.

FAQ

What is an LLM router?

A service that sits between LLM consumers (apps, agents, scripts) and one or more backends (cloud APIs, local models) and decides where each request goes. It picks on task type, cost, speed, remaining quota, model capability, and in this design a privacy level the caller declares. From the consumer’s side it is one OpenAI-compatible address hiding the backend zoo behind it.

What is the difference between an LLM router and an LLM gateway?

A gateway handles transport: authentication, rate limiting, logging, retries. A router adds backend selection on top. Tools like LiteLLM, OpenRouter, and the NVIDIA Inference Blueprint blur the line, since most gateways now ship some selection logic. This project is firmly router-side: every request gets a task type and a privacy tier before any backend is picked.

Why not just use OpenRouter or LiteLLM?

For most people, do. The pattern here is portable: tag your LiteLLM deployments local and cloud, pass the tag from the call site, and you have most of it. Two settings then carry the guarantee, and both default the unsafe way. Give the untagged request nowhere to land rather than every healthy deployment, and disable fallbacks on any key touching private data.

Why prefix the task type instead of adding a separate privacy field?

A separate field can be forgotten, and any default it carries is how leaks happen. The prefix makes the trust tier visible at every call site, log line, error, and code review. local_classifyextract reads as “I want this private.” A privacy field two lines later reads as ceremony. The prefix costs a few characters, and paying that cost is what makes an accidental leak hard.

What happens when a private prompt accidentally gets routed to the cloud?

There is no code path left that can do it. The interesting case is the one people forget: a local_ request arriving when every local machine is saturated. It fails with an error rather than borrowing a cloud backend, because the local half of the routing table contains no cloud target to borrow. An overloaded private tier is an outage, and an outage is correct.

Is task-based routing the same as semantic routing?

No. Semantic routing reads the prompt’s content with a small classifier model and picks a backend from what the prompt looks like it is doing. Task-based routing makes the caller declare the work category up front. Semantic routing is more flexible at the cost of a runtime classifier and silent-misclassification bugs. Task-based routing has zero ambiguity at the boundary.