Keep LLM traffic under control with a self-hosted LLM gateway
When every team calls language models on its own, provider keys spread, token spend has no limit and prompts leave without rules. G8 sends that traffic through one gateway in your infrastructure, with fallback, token limits and prompt rules.
Language models spread faster than the rules around them
Each application that calls a model directly brings its own key, its own provider and its own idea of what a prompt may contain.
Keys in every application
Provider keys are copied into each application that calls a model, and nobody knows where they all are.
With G8Applications call a Route with their own credentials. The provider key is set once, at the gateway.
Token spend without a limit
A loop or a busy feature can use the budget before anyone notices.
With G8Token budgets per Route, provider instance or key are enforced at the gateway.
Prompts without rules
Anything can go into a prompt, including data that shouldn’t leave.
With G8Prompt guard refuses prompts that match your deny patterns, and decorators add your instructions.
One provider, one outage
When the only provider fails or rate limits you, the feature stops.
With G8Fallback moves a request to another provider instance.
One LLM gateway for every model call
Each control is a Plugin on the Route your applications call. They run in the order the Route’s request path shows.
Send each request to the right provider
Route traffic to OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock, Google Gemini and more, or to an OpenAI-compatible endpoint you run yourself. Balance across instances and fall back when one fails.
- FallbackMove a request to another instance when one returns a server error, is rate limited or fails its health checks.
Budget tokens, not just requests
Limit the tokens that traffic to a model can use in a time window, counted as total, prompt or completion tokens. Set one budget for the Route, or one for each provider instance or key.
- On each node or sharedCount on each gateway node, or share the count across nodes in Redis.
Your rules around every prompt
Refuse prompts that match a deny pattern, build prompts from your own templates and put your instructions before or after the user’s messages.
- What the guard readsMatch the last message or the whole conversation.
Provider keys stay at the gateway
The provider key is set in the Plugin and stored on the engine. Roles that can’t manage the Route see it masked, and requests to models are recorded like any other request.
- Data leaves only for what you configurePrompts leave your infrastructure only for the providers you set. A model you run behind an OpenAI-compatible endpoint keeps them inside.
Several teams, one LLM gateway
A typical setup, drawn to show how the parts fit together. It doesn’t describe a specific customer.
Support, sales and engineering teams each built tools that call a language model. Each tool had its own key and its own provider.
Each tool calls its own Route with ai-proxy-multi, a primary provider and a fallback.
ai-rate-limiting gives each team’s Consumer its own token budget.
ai-prompt-guard refuses prompts with card numbers or passwords, and changes to these Routes are recorded in the audit log.