Your LLM traffic and your AI agents, behind the same gateway as your APIs
Send requests to OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock, Google Gemini and more through one gateway in your own infrastructure, with fallback, token limits and prompt rules. Serve your APIs to AI agents with OpenAPI to MCP.
In every edition, Community included
Serve your existing APIs to AI agents
Add the openapi-to-mcp Plugin to a Route and point it at an API’s OpenAPI document. MCP clients, such as AI agents, can then list the API’s operations as tools and call them. The G8 engine does the conversion, inside your infrastructure.
Start from the document you have
G8 reads the OpenAPI document from the address you give. The tools MCP clients see come from the operations it describes.
- Nothing to rewriteThe API itself doesn’t change. Its operations are called at the service base URL you set.
Set it up in one form
The openapi-to-mcp form has two parts: where the document and the service are, and how MCP clients connect to the Route.
- Transportsse or streamable_http, the two ways an MCP client can exchange messages with the Route.
- HeadersHeaders sent with every call to the service, such as a tenant name.
Agents call your API as tools
An MCP client connected to the Route lists the tools and calls them. Each call reaches your API at the service base URL, through the same gateway.
- Inside your infrastructureThe G8 engine does the conversion. No separate MCP server to run.
Your identity, your limits, your logs
The MCP endpoint is a Route like any other. The Plugins you already use run on it, before openapi-to-mcp.
Your identity
Put an authentication Plugin on the Route, and MCP clients sign in like any other client.
Your limits
Rate limits on the Route count MCP requests like any other requests.
Your logs
MCP requests appear in Traffic Explorer and in the logs you already send.
Providers, token limits and prompt rules for every model call
Each capability is a Plugin with its own form in the G8 console. Attach them to the Route your applications call. The Route’s request path shows the order they run in.
Send each request to the right provider
Route traffic to OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock, Google Gemini, Vertex AI, DeepSeek, OpenRouter and AI/ML API, or to any OpenAI-compatible endpoint, including a model you run yourself.
- Keys stay at the gatewayProvider keys are set in the Plugin, masked for roles that can’t manage it.
- BalancingSpread requests across instances by round robin, by a hash of a request value or by how similar prompts are.
- FallbackMove a request to another instance when one returns a server error or is rate limited.
Budget tokens, not just requests
Limit the tokens that traffic to a model can use in a time window, counted as total, prompt or completion tokens or by an expression of your own.
- Per Route, provider or keyOne budget for the Route or for each provider instance, or a budget for each key.
- Settings in the formLimit, Time window, Rules, Instances, Limit strategy, Policy and Rejected code.
- On each node or sharedCount on each gateway node, or share the count across nodes in Redis.
Your rules around every prompt
Allow or refuse prompts by patterns you define, build prompts from your own templates and add your instructions before or after the user’s prompt.
- Prompt guardRefuse prompts that match a deny pattern, or let through only those that match an allow pattern.
- What is checkedOnly the user’s messages, or every role and the whole conversation history.
- DecoratorsPut your messages before or after the client’s, such as a system instruction.
Answer repeated prompts from a cache
Answer a prompt from the cache when it matches an earlier one exactly or, with the semantic layer, by meaning, instead of sending it to the model again.
- Settings in the formLayers, Exact layer, Policy, Semantic layer, Cache key and Bypass rules.
- Needs RedisThe cache is kept in Redis. The semantic layer also needs the RediSearch module.
- Semantic layerAn embedding model and a vector index decide when two prompts mean the same thing.
Let a model rewrite the request
Send the request body to a model with your instruction, and pass the model’s answer on in place of the original body.
- Your model, your instructionChoose the provider and the model that does the rewriting, and give it the instruction.
- Key in the PluginThe provider key is set here and masked for roles that can’t manage it.
- Settings in the formProvider, Auth, Prompt, Options and Override.
Add your documents to the prompt
Search your documents for the request’s question and add what the search returns to the prompt before it reaches the model. Connects to an external service.
- EmbeddingsThe request’s question is embedded with Azure OpenAI.
- Keys maskedBoth keys are stored on the engine and shown only to roles that manage the object.
- Azure AI SearchThe vector search runs in Azure AI Search; what it returns is added to the prompt.
Check content before it reaches a model
Send request content, and the model’s response if you choose, to a moderation service and refuse what scores above your thresholds. Connects to an external service.
- Amazon ComprehendScores toxicity overall and by category, such as hate speech, insult or violence.
- Request, response or bothCheck what the client sends, what the model answers, or both.
- The code you chooseA refused request is answered with the status you set.
Screen prompts with Lakera Guard
Check prompts, model responses or both with Lakera Guard. Block what it flags, or let it through and log it. Connects to an external service.
- Your Lakera projectThe project you name decides which detectors and thresholds apply.
- Settings in the formAPI key, Project ID, Direction, Action, Deny code and Fail open.
- When Lakera can’t be reachedChoose whether requests go through or are denied.
Where your keys and prompts go
- Provider keys stay at the gatewayApplications call the Route with their own credentials. The provider key is set in the Plugin and stored on the engine, and roles that can’t manage the Route see it masked.
- Model calls recorded like any requestRequests to models appear in Traffic Explorer like any other request, with the masking you set. The Plugin can add the model, the duration and the token counts, or the request and response bodies, to the records your logger Plugins write.
- Data leaves only for what you configurePrompts leave your infrastructure only for the providers and services you set in a Plugin: a model provider, an embedding or search service, a moderation service or Lakera Guard. A model you run behind an OpenAI-compatible endpoint keeps them inside.
AI traffic runs on the same Routes, roles, Environments and audit record as your APIs
Identity Plugins, rate limits and logging work on a Route to a model as on any other Route. The G8 engine checks each change to it, and G8 records the change before it’s applied.
What the AI capabilities connect to
AI Gateway runs on the same G8 engine clusters as your APIs. Each capability needs only what it connects to, and only when you use it.
An account and a key for each model provider you send traffic to, or an OpenAI-compatible endpoint you run.
For semantic and exact caching, and for token limits shared across gateway nodes. The semantic layer needs the RediSearch module and an embedding model from OpenAI or Azure OpenAI.
For retrieval: Azure AI Search, with embeddings from Azure OpenAI.
For content moderation, such as Amazon Comprehend.
For Lakera Guard: the project and its API key.