# Why the agent fleet runs behind one gateway > My agents each used to call the model provider directly, with their own key. A model gateway changed that: one shared doorway every request passes through, like an office routing all its post through a single mailroom. Now every agent calls one address, so the keys, the model and API choice, the spend and the traces collect in one box, and jobs that were painful per server I do once. When [I swapped the bloated harness for a leaner one](/garden/openclaw-analyst-workflows), the agents got easier to reason about. But they were still scattered: separate [Lightsail servers](/garden/lightsail-vps), each holding its own credential and calling Amazon Bedrock on its own. The layout I landed on is boring on purpose. A gateway sits between the agents and everything external, models and data APIs alike, and a whole set of capabilities falls out of that one seam. ![Agent fleet architecture. The agent servers, one to many NanoClaw servers that each hold only a placeholder key and can scale out, call one endpoint, the model gateway running LiteLLM. LiteLLM holds every real credential in the fleet and is its single egress. It fronts the model providers Anthropic, OpenAI, Google Gemini and Amazon Bedrock, and fronts external data APIs such as data.gov.sg and the YouTube API through pass-through routes, so those keys rotate in one place too. The gateway ships every call's trace to the observability host, which runs Langfuse, Prometheus and Grafana for traces, metrics, spend, evals and optimisation, with no agent instrumented.](/garden-assets/experience/atom_notes/model-gateway/model-gateway-architecture.svg?v=3adb08234b) The whole layout in one view. The agent servers call one endpoint, and the gateway fronts every model provider and external API behind it, holds all the real keys, and passes model output and data back. Every call's trace lands on the observability host. ## What each part does Three roles, split across a few cloud servers on one private network. * **The agent servers (NanoClaw #1, #2, and more).** Where the agents run. Each is a cloud server running an agent harness such as NanoClaw and nothing else, and I add more when the work needs them. None of them holds a real key. Every one calls a single endpoint, the gateway, for every model call and every external API. * **The model gateway (LiteLLM).** One small box every outbound call passes through. It fronts the model providers (Anthropic, OpenAI, Google Gemini, Amazon Bedrock) behind a single endpoint, so [an agent names a model](/garden/subagent-brief) and never learns which provider answered. Through pass-through routes it fronts external data APIs the same way (data.gov.sg, the YouTube API), so their keys live here too. Every real credential in the fleet lives in this one box, and a Postgres behind it holds each agent's virtual key, its budget and its spend. * **The observability host.** Runs Langfuse, Prometheus and Grafana. The gateway ships every call's prompt, response, tokens, cost and latency here through one callback, so no agent needs instrumenting. Traces, metrics and spend land here, and it is where evals and optimisation run. ## The gateway is the seam that buys the rest Point the agents at one endpoint instead of at every provider and API, and the capabilities stop being per-agent chores: | Capability | What the seam buys | | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Key rotation | Every real key, models and data APIs alike, lives only in the gateway; agents hold a placeholder, so I rotate on a schedule or by hand and redeploy nothing. A key that leaks or ages is a standing risk, so rotation stays routine. Where a provider takes several keys, the gateway spreads calls across them round-robin, so no one key's rate limit caps the fleet and pulling a retired key never drops traffic. | | Provider swap and fallback | Agents ask for a model name, not a provider. Move a role from Anthropic to Gemini, OpenAI or Bedrock, and fall back when one is down, without touching agent code. | | Budgets and limits | Each agent gets a virtual key with its own budget and rate limit, so a runaway loop or a leaked key is capped, not a surprise bill. | | Per-agent routing | Which model each agent gets is a gateway setting, so \[\[model-per-agent | | One egress | Every outbound call, model or API, leaves through one box on one port, so the firewall to reason about is tiny. | | Observability for free | One callback logs every call. The agents stay ignorant of it, and the spend and latency of the whole fleet land in one place. | ## The honest limit It is deployed and carrying calls, but I am not using all of it yet. The evals, the guardrails and the cost optimisation are set up more than they are used. The SaaS credential rotation and a [smart router](/garden/smart-router-prompt-cache) are still experiments, not settled. And the seam cuts both ways: routing every call through one box makes that box a single point of failure. Splitting the roles apart narrowed the blast radius, it did not remove it. Recovery is one setting, repoint the agents straight at a provider, so an outage is easy to recover from rather than fatal. ## The bottom line The upgrade wasn't a better model, it was the seam. Put a gateway between the agents and everything external, and key rotation, provider swap, routing, budgets and cost tracking all move from things you do per agent to things you do once. The agents get simpler because they only know one endpoint, and everything that is annoying to secure or change collects in one box.