I Built a Routing Layer So My AI Agent Would Stop Bleeding Money
So here is the problem I started with! I had my Hermes Agent hitting OpenRouter directly, with one model hardcoded for every single request. I had to manually change models and figure out the best one to use all by myself. Then I added cron jobs. Now the same handful of prompts were firing on a schedule, and my API spend started climbing in a straight line with absolutely zero intelligence behind it.
So naturally as one does, I built a routing and caching layer that sits between the agent and OpenRouter. It absolutely did not work the way I planned. The original design broke multiple times, but the thing that actually shipped turned out to be way more interesting than the original plan. This post is the deep dive into the full topology, the classifier problem in detail, and exactly how the "default model" trick keeps my costs down.
The Topology: Three Processes, Not One
Nothing here is one monolithic proxy. Much like the modular monoliths I love building, it is three separate, wonderfully boring processes, each doing exactly one job:
- Hermes agent sends every chat completion to port
4000, authenticated with a master key. - Classifier proxy looks at the model field on the incoming request and decide whether to leave it alone or rewrite it.
- LiteLLM proxy on port
4001does the actual routing. It holds the tier definitions, talks to Redis for caching, and forwards everything to OpenRouter.
💡Wait, what exactly is LiteLLM?
If you have not played with it yet, think of LiteLLM as the ultimate universal translator for AI models! Instead of writing custom API calls for OpenAI, Anthropic, Gemini, and OpenRouter, LiteLLM sits in the middle and lets you talk to all of them using the exact same standard OpenAI format. In our setup, it is acting as the heavy lifter for the proxy layer by seamlessly handling the routing to different model tiers, managing fallbacks when things break, and chatting with Redis so we do not pay twice for the exact same cron job prompt. It basically makes infrastructure boring again, which is exactly what you want!
Why use LLMLite when OpenRoutes unifies AI models? Two words - vendor lock-in :)
Redis and OpenRouter sit safely behind LiteLLM, entirely out of sight from the classifier. The classifier proxy never touches a cache or a provider directly. It only ever talks to LiteLLM.
This split exists because of a massive failure on my part, which is absolutely worth explaining properly because it is the part of this build that really burned my time!
The Classifier Problem In Depth
Plan A: A LiteLLM Callback Hook The original design did not need a second process at all. LiteLLM Proxy supports custom callbacks where you write a custom script, register an async pre call hook, and that hook gets a chance to inspect and rewrite every request before it goes anywhere. Cheap classification, done entirely inside LiteLLM, one process total.
Spoiler alert: it does not work. Not "works with caveats" but straight up does not fire. LiteLLM version 1.91 does not dispatch that hook for custom callbacks the way the documentation implies (bug open on GitHub). I wrote the hook, registered it, sent requests through, and watched them sail past completely untouched. No error. No warning in the logs. The callback just sits there, registered and inert, while every request bypasses it entirely. I burned a chunk of my day convincing myself this was not a config mistake on my end before I finally accepted it was a framework bug and moved on.
Plan B: Pull the Logic in Front of LiteLLM If you cannot hook the inside of the box, do not fight it. Just put your own box in front of it! That is the entire reason the classifier proxy exists. It is a dumb, standard library only WSGI server on port 4000 that intercepts the request before LiteLLM ever sees it, decides what to do, and forwards accordingly. LiteLLM goes back to doing what it is actually good at doing (caching, model groups, fallback chains) and stops being asked to handle request time decision logic it structurally cannot do right now.
The Classification
My very first version of the classifier relied on regex heuristics on the prompt text to look for keywords like "refactor" or "fix typo" and bucket them accordingly. I ripped it out almost immediately. Regex is either too aggressive and flags things it should not, or it silently misses anything phrased in a weird way it did not anticipate. A coding agent is definitely not using a controlled vocabulary. There is no keyword list that reliably separates "rename this variable" from "redesign this entire module."
So the classifier itself became a second, smaller LLM call!
Only the last user message goes to the classifier. No tool definitions, no history, and no system prompt bloat. Just a fixed instruction to return exactly one JSON object with a tier of SIMPLE, MEDIUM, COMPLEX, or REASONING, at temperature 0. This is a very simple model for classification can this can be engineered to the nth level. This simple classifier fits my use case for a simple hermes agents, so I'm willing to live with it :)
That is an LLM call in the critical path of every single sentinel request, which means it is also a shiny new thing that can fail. And boy, does it fail in more ways than you would expect from something that just has to return four words in a JSON wrapper.
Every one of these failure modes is real and has actually happened to me:
- Timeout: The classifier model i started with is a free tier Nemotron model from OpenRouter. Free tiers queue under heavy load, and 30 seconds sometimes just is not enough.
- HTTP Error: Thrown from LiteLLM itself, which is logged distinctly so it is not confused with a model level failure.
- Empty or Unparseable JSON: Reasoning style models sometimes want to "think" before answering, and if you cap max tokens too low, you get truncated output that is not valid JSON. This one actually bit me. My first attempt used max tokens set to 20, and the model would get cut off mid thought before it ever emitted the JSON. It is bumped to 100 now, with a lighter LLM now :)
- Unknown Tier Label: The model returns something that is not one of the four expected strings.
- Recursion Guard Tripped: This one is structural and totally wild. See below!
The important design decision here is not the list of failure modes. It is what happens next, which is absolutely no retries. The moment any of these fires, the request gets rewritten straight to a fixed, cheap fallback model like Deepseek-v4-flash and moves on. I did not want a classifier failure to become a classifier retry storm under load. The whole point of this layer is to reduce spend and latency, not add a second point where a struggling free tier model can make things much worse.
The Recursion Guard
Once you put a proxy in front of a proxy, you have created the magical possibility that the proxy calls itself. The classifier needs to make its own LLM call to figure out the tier. If that sub call were routed back through port 4000, it would hit the sentinel check again, try to classify that request, and potentially loop into infinity. Really got stuck into an infinte loop and laughed way harder than I needed to when i realized this.
I built two independent protections against this, just to be safe:
- Architectural: The classifier sub call goes directly to localhost port 4001 for LiteLLM, never back through port 4000. The proxy that would create the loop is structurally incapable of receiving its own sub request.
- Thread Local Flag: This acts as a second layer, just in case the architecture is ever changed. If the classification function is somehow re entered on the same thread while already active, it immediately returns None, which routes into the same hard fallback path described above.
Belt and suspenders, folks. A silent infinite loop in a proxy is a much worse failure than a slightly too cheap model answering your question.
How the Cheap Default Actually Saves Money
This is the part that is super easy to describe wrong, so here is the precise mechanism.
OpenRouter has no model literally called "Auto". But Hermes needs a real, valid, selectable model string sitting in its config at all times because its runtime will not let you configure "figure it out yourself." So the fix was not to invent a fake routing keyword. It was to point Hermes at a real model that happens to be free, and give that one specific model string a second meaning: "route me."
Concretely, in my setup:
model:
default: nvidia/nemotron 3 ultra 550b a55b free
provider: custom litellm router
That specific Nemotron model was chosen deliberately. It is a confirmed free tier OpenRouter model with a 1 million token context window and zero cost for tokens in and out. Hermes always sends this exact string as the model. Cost only enters the picture in the step after that! The classifier reads the actual content of the request and reassigns it to whichever of the four ascending tiers matches the work:
Tier Simple (formatting, typos, renames) moves up to Tier Medium (general coding, multiple step tasks) moves up to Tier Complex (architecture, big refactors) moves up to Tier Reasoning (proofs, root cause debugging).
The savings compound in three separate places, and it is worth being precise about which one does what:
- The default is free, and nothing defaults to expensive. Simple tasks get routed to Tier Simple, which itself uses cheap or free models like llama 4 scout or deepseek v4 flash. The agent never pays reasoning model prices for a variable rename, because the entry point costs nothing and every step up the tier ladder has to be explicitly earned by the classifier deciding the task genuinely needs it.
- Even failure is cheap. When the classifier itself breaks, the hard fallback is not a random guess or an expensive safety net. It is deepseek v4 flash, the same low cost model used inside Tier Simple. A broken classifier degrades to a cheap model for everything rather than an expensive model to be safe. That was a deliberate choice because uncertainty should never resolve toward spending more money.
- Redis eliminates the cost of repeats entirely. This matters most for the workload that started this whole project: Exact match caching means the same model, same messages, and same parameters within that window returns instantly from Redis with zero OpenRouter calls. Measured on this setup, the first request took about
15,523ms, and the second identical request took about31ms!That is not a discount on the second call. It is a zero dollar call!
The override path matters here too, and it is the same mechanism from the routing diagram earlier. If you or Hermes or a test script send anything other than the sentinel string, like asking for Tier Complex or a raw OpenRouter slug directly, the classifier does nothing and forwards it exactly as is. The cheap by default behavior is specifically a property of the sentinel value, not something baked into every request. You can always explicitly ask for the expensive model and get it immediately with no classification tax and no override to fight.
What I Did Not Build and Why
Every one of these got cut for the exact same reason. This entire setup runs on a tiny box with 3.7 GB of RAM of my home server, not some massive lab machine. Sure, when I am doing heavy agentic usage, I am absolutely maxing out the 16GB limit on my M4 Mac, but for this cloud layer, we are playing on hard mode with memory constraints.
- No semantic caching! A Qdrant or embedding based cache would catch near duplicate prompts, not just exact ones, but it adds real memory and real risk. A "close enough" match returning a wrong answer for a cron job is way worse than a cache miss. Exact match Redis was enough because my cron prompts repeat verbatim.
- No local model weights anywhere. Classification is an API call to a free tier model, not on box inference. Running weights on 4 GB of RAM was never viable.
- No LiteLLM spend database. The OpenRouter dashboard already answers "what did I spend," which is the question that actually matters day to day.
- No containerizing LiteLLM or the classifier. Both run as bare nohup processes with a reboot crontab entry. Docker is reserved for exactly one thing: Redis, capped at 128MB with LRU eviction, because that is the one component where an unbounded memory footprint would actually be dangerous on this hardware.
The One Design Decision I Would Keep If I Rebuilt This From Scratch
Not the LLM classifier. Not the tier structure. The thing that mattered most was making the override path free and instant. Any request that is not the exact sentinel string skips the entire routing and classification machinery and goes straight through. Every other layer in this system (the classifier, the tiers, the cache) is allowed to fail, time out, or misbehave, because there is always a direct, zero overhead path around all of it. That is the property that let me build the riskier parts (an LLM deciding cost tiers or a standard library proxy standing in for a broken framework hook) without worrying that a bug in them would take the whole agent down. Keep it resilient, keep it cheap, and keep shipping!
Related Projects
Agentic Knowledge Assistant
An LLM-powered, multi-channel assistant that uses Retrieval-Augmented Generation (RAG) to autonomously answer employee o...
Autonomous Content-to-Learning Engine
An AI system that ingests PDFs, videos, or documents and autonomously creates assessments, flashcards, and learning summ...
Embeddable Role-Aware Chat Widget
A lightweight AI widget that plugs into any platform and adapts answers dynamically based on user role and platform cont...