Skip to main content

AI Gateway v2

Stacklok Enterprise

AI Gateway v2 is a component of Stacklok Enterprise.

Learn more about Stacklok Enterprise.

AI Gateway v2 is a self-hosted control point between AI clients and large language model (LLM) providers. It authenticates each caller, routes the request to the provider that serves the model, and applies identity, budget, data protection, and audit policies on the way through.

AI Gateway v2 passes each request to the provider in the API format the client sent and doesn't translate between APIs. Clients use the provider's own API, such as OpenAI Chat Completions, the OpenAI Responses API, Anthropic Messages, or the Gemini API, and the gateway serves each model on the APIs its provider speaks.

What you can do​

  • Set cost budgets for users and groups.
  • Limit which models each user or group can call, and show each caller only the models they can use.
  • Screen requests for prompt injection, payment card data, and personally identifiable information (PII).
  • Record structured audit events and forward them to a security information and event management (SIEM) system.
  • Authenticate callers with your identity provider or with virtual API keys, while provider credentials stay in the gateway.
  • Export traces and metrics to an OpenTelemetry collector, attributed to the users and groups that sent the traffic.

How configuration works​

You configure AI Gateway v2 through Helm values under the ai-gateway-v2 key of the platform chart. The chart renders the values into a configuration file in a ConfigMap and mounts provider credentials from Kubernetes Secrets.

A configuration or credential change takes effect through a pod rollout. The rollout keeps the previous pods serving until the new pods are ready, so a configuration the gateway rejects at startup never removes serving capacity. Pods can still become ready with a wrong budgets or provider setting, so send an authenticated test request after each rollout. AI Gateway v2 has no custom resources and no management API.

Manage budgets in the console or through the Enterprise Manager API. See Manage budgets and Budgets and pricing.

How AI Gateway v2 relates to the AI Gateway​

AI Gateway v2 and the Envoy-based AI Gateway are two implementations of the same product. They share the console, budgets, pricing, directory, and virtual API keys, but each has its own configuration. A platform runs one of them, except during a migration window. To move from the AI Gateway to AI Gateway v2, see Migrate from the AI Gateway.

Before you start​

Install AI Gateway v2 before applying the examples in this section. See Configure AI Gateway v2.

Examples on these pages show values relative to the platform chart's values file and use <NAMESPACE> for the namespace the platform is installed into, <RELEASE_NAME> for the Helm release name, and <CHART> for the platform chart reference you install from. In prose, value paths are relative to the ai-gateway-v2 key, except those that start with global. or another chart's key, such as enterprise-manager..

For the client-facing endpoints, credential headers, and error responses, see the endpoints reference.

Contents​