# LLM Gateway

`llmgatewayText` calls one OpenAI-compatible endpoint that routes across providers, falls back on failure, and records usage and cost.

## 1. Install

```sh
npm i @tanstack/ai-llmgateway
```

## 2. Set the API key

Get a key from the [LLM Gateway dashboard](https://llmgateway.io). Put it in the environment of your server:

```bash
LLM_GATEWAY_API_KEY=llmgtwy_your-api-key
```

`llmGatewayText` and `llmGatewaySummarize` read `LLM_GATEWAY_API_KEY`. To pass the key in code, use the `create*` factory of the same adapter:

```typescript
import { createLLMGatewayText } from "@tanstack/ai-llmgateway";

const adapter = createLLMGatewayText("gpt-5.6-terra", process.env.LLM_GATEWAY_API_KEY!);
```

## 3. Stream a chat from the server

```typescript
import {
  chat,
  chatParamsFromRequest,
  toServerSentEventsResponse,
} from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

export async function POST(request: Request) {
  const { messages, threadId, runId } = await chatParamsFromRequest(request);

  const stream = chat({
    adapter: llmGatewayText("gpt-5.6-terra"),
    messages,
    threadId,
    runId,
  });

  return toServerSentEventsResponse(stream);
}
```

## 4. Render the chat

The client is the same for every provider. Point `useChat` at your route:

```tsx
import { useState } from "react";
import { useChat, fetchServerSentEvents } from "@tanstack/ai-react";

export function Chat() {
  const [input, setInput] = useState("");
  const { messages, sendMessage } = useChat({
    connection: fetchServerSentEvents("/api/chat"),
    threadId: "support-chat",
  });

  return (
    <form
      onSubmit={(event) => {
        event.preventDefault();
        sendMessage(input);
        setInput("");
      }}
    >
      {messages.map((message) => (
        <p key={message.id}>
          {message.parts
            .map((part) => (part.type === "text" ? part.content : ""))
            .join("")}
        </p>
      ))}
      <input value={input} onChange={(event) => setInput(event.target.value)} />
    </form>
  );
}
```

For Vue, Solid, Svelte, Angular, and the other frameworks, see [Quick Start](../getting-started/quick-start#3-render-the-chat).

You now have a streaming chat through LLM Gateway. To try another model, change the id, for example to `claude-opus-5` or `kimi-k3`.

## Pick a model and a provider

Any model on [llmgateway.io/models](https://llmgateway.io/models) works. Pass its id as the model name:

- A bare id lets the gateway pick the best provider that is available.
- A `provider/` prefix always sends the request to that provider.

```text
model: "gpt-5.6-terra"          // the gateway picks the provider
model: "claude-sonnet-5"        // the gateway picks the provider
model: "moonshot/kimi-k3"       // always sent to Moonshot
model: "fireworks/kimi-k3"      // always sent to Fireworks
```

## Tools

Tools work the same as with every adapter:

```typescript
import {
  chat,
  chatParamsFromRequest,
  toServerSentEventsResponse,
  toolDefinition,
} from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";
import { z } from "zod";

const getWeatherDef = toolDefinition({
  name: "get_weather",
  description: "Get the current weather",
  inputSchema: z.object({
    location: z.string(),
  }),
});

const getWeather = getWeatherDef.server(async ({ location }) => {
  return { location, temperature: 72, conditions: "sunny" };
});

export async function POST(request: Request) {
  const { messages, threadId, runId } = await chatParamsFromRequest(request);

  const stream = chat({
    adapter: llmGatewayText("gpt-5.6-terra"),
    messages,
    threadId,
    runId,
    tools: [getWeather],
  });

  return toServerSentEventsResponse(stream);
}
```

See [Tools](../tools/tools).

## Summarization

```typescript
import { summarize } from "@tanstack/ai";
import { llmGatewaySummarize } from "@tanstack/ai-llmgateway";

const result = await summarize({
  adapter: llmGatewaySummarize("gpt-5.4-mini"),
  text: "Long article text...",
  threadId: "thread-1",
  stream: false,
});

console.log(result.summary);
```

## Models and options

### Model options

The gateway takes the standard Chat Completions parameters and sends them to the provider that it picks. The gateway removes the parameters that a provider does not support. Sampling options go in `modelOptions`:

```typescript
import { chat } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

const stream = chat({
  adapter: llmGatewayText("kimi-k3"),
  messages: [{ role: "user", content: "Hello!" }],
  threadId: "thread-1",
  modelOptions: {
    temperature: 0.7,
    max_completion_tokens: 4096,
  },
});
```

### Reasoning

Set `reasoning` on `chat()`:

```typescript
import { chat } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

const stream = chat({
  adapter: llmGatewayText("kimi-k3"),
  messages: [{ role: "user", content: "Plan a database migration." }],
  threadId: "thread-1",
  reasoning: "high",
});
```

It goes out as `reasoning_effort`. The types take the levels of that model from the LLM Gateway catalog. No model takes `budgetTokens`.

Reasoning models stream their thinking as `reasoning_content` deltas. The adapter turns them into AG-UI `REASONING_*` events. See [Reasoning](../chat/reasoning).

### Models

`LLMGATEWAY_CHAT_MODELS` is the chat model list from the LLM Gateway catalog. Each model has type data for its input kinds and its reasoning levels. Autocomplete in the editor uses this list.

An id that is not in the list still works. It gets text-only input and the generic options.

`adapter.inputModalities` lists the input kinds of the selected model. It is `undefined` for a model that this package does not know. See [Check what a model accepts](../advanced/extend-adapter#check-what-a-model-accepts).

## Advanced

### Self-hosted gateway

To use your own deployment, set `baseURL`:

```typescript
import { createLLMGatewayText } from "@tanstack/ai-llmgateway";

const adapter = createLLMGatewayText("gpt-5.6-terra", process.env.LLM_GATEWAY_API_KEY!, {
  baseURL: "https://api.llmgateway.io/v1", // the hosted default. Set your own URL here.
});
```

The adapter works the same against a self-hosted gateway.

Need a header on each request, or a log of each model call? This adapter supports [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call).

### API reference

| Factories | Creates |
| --- | --- |
| `llmGatewayText(model, config?)` / `createLLMGatewayText(model, apiKey, config?)` | Chat adapter |
| `llmGatewaySummarize(model, config?)` / `createLLMGatewaySummarize(model, apiKey, config?)` | Summarization adapter |

The short factory reads `LLM_GATEWAY_API_KEY`. The `create*` factory takes an explicit API key.