AI platform lab · 2026
Testing an LLM gateway's controls instead of trusting its config
When AI coding agents send every model call through one gateway, that gateway enforces budgets, model access and cost reporting. I built a LiteLLM lab with production-style settings (retries, fallback, a cache, budgets, MCP access and spend tags) and checked what each control actually does. Most did what their configuration said. Two didn't.
Context
An LLM gateway sits between engineers' tools and the model providers. It issues each person a virtual key, exposes tools to agents through the Model Context Protocol (MCP), groups keys into teams with budgets and allowed models, retries and falls back when a provider fails, caches repeat requests, and logs spend. Teams rely on those settings without seeing them work. I wanted to test the behavior, not read the YAML.
What worked as configured
- Fallback across providers. With the primary Claude deployment pointed at a dead endpoint, requests still succeeded: Gemini answered in about seven seconds, most of it spent waiting on the failed primary, with the fallback adding about two. The response names the model that answered, so anything that logs or bills by model has to read that field.
- Response cache. An identical request came back about 25× faster from Redis and logged zero cost. The cost header still showed the full price, so spend reports have to come from the logs, not the headers. I made caching opt-in per request, because caching every identical prompt can hide model changes and skew budget tests.
- Key budgets and model allowlists. The gateway cut off a low-budget key on its third call and refused a model outside its allowlist. Because the gateway checks budgets before a call and calculates cost after, the cut-off can overshoot by one request. Dashboards lagged enforcement because the gateway writes spend to the database in batches.
- MCP access. With "allow all keys" off, a key could reach an MCP server only through a team that owned it. The gateway even refused to grant the server to a key directly.
Two gaps that config review wouldn't catch
- A custom auth hook switched off team-level controls.
- I added a small auth hook that lets MCP clients that can't set headers pass their key in the URL. With that hook enabled and one setting (
custom_auth_run_common_checks) left at its default, the gateway stopped enforcing team blocks, team budgets and team model allowlists, for ordinary chat calls as well as MCP calls. Key-level controls still worked. - LiteLLM's startup warning mentions budgets and allowlists. It doesn't mention that blocked teams also get through, and in the lab the bypass hit header-authenticated chat calls too, not only the URL-key path.
- I compared three setups side by side: no hook, the hook, and the hook with the setting on. Setting it to
truerestored all three controls and kept the URL-key behavior.
- I added a small auth hook that lets MCP clients that can't set headers pass their key in the URL. With that hook enabled and one setting (
- One API route dropped cost tags.
- Tags that say which tool or component made a call are how a gateway splits spend.
- A client calling the gateway's OpenAI-style endpoint had its tags recorded.
- A client calling the Anthropic-style endpoint sent the same tags in a field that route ignores, so the gateway recorded none of them. Nothing errors, and the spend report just loses that dimension.
Smaller lessons
- Wait out the cache before trusting a result. Blocking a team takes effect only after the gateway's auth cache expires. My first comparison, with a two-second wait, showed a blocked team still working in every setup, which was the cache, not the hook. Re-running with a 70-second wait, past the roughly one-minute cache window, gave the real result.
- Read the upgrade, not just the changelog. Moving the lab to a newer LiteLLM release applied nearly a hundred schema migrations cleanly. It also changed the budget error from HTTP 429 to 422, which matters to any client that retries on 429. Embeddings that had failed started working, and wildcard models stopped being listed.
- Agent runs are mostly fixed overhead. A coding agent's first call carried tens of thousands of tokens of instructions and tool definitions before any user text. In a multi-turn run, prompt caching made later turns 80–87% cheaper than the first, despite more tokens.
My part: I built the sandbox (a LiteLLM gateway and Postgres in Docker Compose), set the goal of a production-style configuration, directed the investigation, ran the interactive client tests and reviewed every result. An AI coding agent (Claude Code) added the Redis cache, upgraded the stack, built the test scripts and ran the matrix under my direction. That's how I work day to day.
What I carry forward
A control you haven't seen fire is an assumption. For anything that enforces money or access, I now want a small test that proves it blocks what it should, re-run after every upgrade, not just a config review.