Skip to content
TanStackAI
Adapters

ElevenLabs

@tanstack/ai-elevenlabs calls ElevenLabs for voice agents, speech, music, sound effects, transcription, and voice design. It has no text chat.

The quick setup below builds a live voice conversation with an ElevenLabs agent.

1. Install

shell
npm i @tanstack/ai @tanstack/ai-client @tanstack/ai-elevenlabs

@tanstack/ai and @tanstack/ai-client are peer dependencies of the adapter. Step 4 also uses the React package, @tanstack/ai-react.

2. Set the API key and the agent

  1. Get an API key from the ElevenLabs dashboard.
  2. In the Conversational AI section of the dashboard, create an agent. Set its voice, personality, knowledge, and tools there.
  3. Put both values in the environment of your server:
shell
ELEVENLABS_API_KEY=your-elevenlabs-api-key
ELEVENLABS_AGENT_ID=your-agent-id

All adapters read ELEVENLABS_API_KEY. If you pass agentId in code, ELEVENLABS_AGENT_ID is optional.

3. Create a voice token on the server

The browser must not see your API key. So the server makes a signed WebSocket URL, and the browser connects with it. The URL is valid for 30 minutes.

ts
import { realtimeToken } from "@tanstack/ai";
import { elevenlabsRealtimeToken } from "@tanstack/ai-elevenlabs";

export async function POST() {
  const token = await realtimeToken({
    adapter: elevenlabsRealtimeToken({
      agentId: process.env.ELEVENLABS_AGENT_ID!,
    }),
  });

  return Response.json(token);
}

Serve this handler at /api/realtime-token.

4. Talk to the agent

useRealtimeChat gets the token, opens the microphone, and plays the voice of the agent:

tsx
import { useRealtimeChat } from "@tanstack/ai-react";
import { elevenlabsRealtime } from "@tanstack/ai-elevenlabs";

export function VoiceChat() {
  const { status, messages, connect, disconnect, pendingAssistantTranscript } =
    useRealtimeChat({
      getToken: () =>
        fetch("/api/realtime-token", { method: "POST" }).then((r) => r.json()),
      adapter: elevenlabsRealtime(),
    });

  return (
    <div>
      <button onClick={status === "idle" ? connect : disconnect}>
        {status === "idle" ? "Start conversation" : "End conversation"}
      </button>
      {messages.map((message) => (
        <p key={message.id}>
          <strong>{message.role}: </strong>
          {message.parts.map((part, index) => (
            <span key={index}>
              {part.type === "text" ? part.content : null}
              {part.type === "audio" ? part.transcript : null}
            </span>
          ))}
        </p>
      ))}
      {pendingAssistantTranscript && <p>AI: {pendingAssistantTranscript}...</p>}
    </div>
  );
}

Click Start conversation and speak. The agent answers out loud, and the transcript shows up below the button. For the other frameworks and for the full realtime API, see Realtime Voice Chat.

Client tools

The agent can call tools that run in the browser. Define them with toolDefinition() and pass them to the hook:

ts
import { toolDefinition } from "@tanstack/ai";
import { useRealtimeChat } from "@tanstack/ai-react";
import { elevenlabsRealtime } from "@tanstack/ai-elevenlabs";
import { z } from "zod";

const getWeatherDef = toolDefinition({
  name: "getWeather",
  description: "Get weather for a location",
  inputSchema: z.object({
    location: z.string(),
  }),
  outputSchema: z.object({
    temperature: z.number(),
    conditions: z.string(),
  }),
});

const getWeather = getWeatherDef.client(async ({ location }) => {
  const res = await fetch(`/api/weather?location=${location}`);
  return res.json();
});

export function useWeatherVoiceChat() {
  return useRealtimeChat({
    getToken: () =>
      fetch("/api/realtime-token", { method: "POST" }).then((r) => r.json()),
    adapter: elevenlabsRealtime(),
    tools: [getWeather],
  });
}

The adapter turns each result into a string and sends it to the agent.

Text-to-speech

elevenlabsSpeech makes one audio clip from text. voice is an ElevenLabs voice id:

ts
import { generateSpeech } from "@tanstack/ai";
import { elevenlabsSpeech } from "@tanstack/ai-elevenlabs";

const result = await generateSpeech({
  adapter: elevenlabsSpeech("eleven_v4"),
  text: "Hello from ElevenLabs!",
  threadId: "thread-1",
  voice: "21m00Tcm4TlvDq8ikWAM",
  format: "mp3",
});

console.log(result.audio); // Base64-encoded audio
  • format takes mp3 (the default), pcm, opus, or wav. The adapter throws for aac and flac before it calls the API.
  • For a dialogue with several voices, or for timings, see Dialogue and timings.

See Text-to-Speech.

Music and sound effects

elevenlabsAudio makes music or a sound effect. The model picks which one:

ts
import { generateAudio } from "@tanstack/ai";
import { elevenlabsAudio } from "@tanstack/ai-elevenlabs";

const music = await generateAudio({
  adapter: elevenlabsAudio("music_v2_5"),
  prompt: "An upbeat synthwave track for a product launch",
  threadId: "thread-1",
});

const sfx = await generateAudio({
  adapter: elevenlabsAudio("eleven_text_to_sound_v2"),
  prompt: "A glass shattering on concrete",
  threadId: "thread-1",
});

console.log(music.audio, sfx.audio);

See Audio Generation.

Transcription

ts
import { generateTranscription } from "@tanstack/ai";
import { elevenlabsTranscription } from "@tanstack/ai-elevenlabs";

const result = await generateTranscription({
  adapter: elevenlabsTranscription("scribe_v2"),
  audio: "https://example.com/recording.mp3",
  threadId: "thread-1",
});

console.log(result.text);

See Transcription.

Voice design

elevenlabsVoiceDesign makes a new voice from a description. Use the voice id that it returns in generateSpeech():

ts
import { generateVoice } from "@tanstack/ai";
import { elevenlabsVoiceDesign } from "@tanstack/ai-elevenlabs";

const result = await generateVoice({
  adapter: elevenlabsVoiceDesign("eleven_ttv_v3"),
  prompt: "A warm, gravelly narrator in his sixties with a slight Irish lilt",
  threadId: "thread-1",
  name: "Irish Narrator",
});

const [voice] = result.voices;
if (!voice) throw new Error("The provider returned no voices.");

console.log(voice.voiceId);
console.log(voice.saved); // true, because a name was given
  • Without a name, you get preview voices to listen to.
  • With a name, ElevenLabs keeps the best voice in your voice library.

For reference audio and the other options, see Voice Creation.

Models and options

ActivityFactoryModels
Text-to-speechelevenlabsSpeecheleven_v4, eleven_v4_turbo, eleven_v3, eleven_v3_conversational, eleven_multilingual_v2, eleven_flash_v2_5, eleven_flash_v2
MusicelevenlabsAudiomusic_v2_5, music_v2
Sound effectselevenlabsAudioeleven_text_to_sound_v2
TranscriptionelevenlabsTranscriptionscribe_v2, scribe_v2_medical
Voice designelevenlabsVoiceDesigneleven_ttv_v3, eleven_multilingual_ttv_v2

The adapter also accepts the deprecated ids eleven_turbo_v2, eleven_turbo_v2_5, eleven_monolingual_v1, scribe_v1, and music_v1. Use the newest model in the same row.

The package exports the lists as ELEVENLABS_TTS_MODELS, ELEVENLABS_AUDIO_MODELS, ELEVENLABS_TRANSCRIPTION_MODELS, and ELEVENLABS_VOICE_MODELS.

Advanced

Agent overrides

To change the agent for one session, set overrides when you make the token. Your dashboard settings do not change:

ts
import { realtimeToken } from "@tanstack/ai";
import { elevenlabsRealtimeToken } from "@tanstack/ai-elevenlabs";

const token = await realtimeToken({
  adapter: elevenlabsRealtimeToken({
    agentId: process.env.ELEVENLABS_AGENT_ID!,
    overrides: {
      voiceId: "custom-voice-id",
      systemPrompt: "You are a helpful voice assistant.",
      firstMessage: "Hello! How can I help you today?",
      language: "en",
    },
  }),
});

console.log(token.expiresAt);
OptionTypeDescription
agentIdstringThe agent id from the dashboard. If you do not set it, the adapter reads ELEVENLABS_AGENT_ID.
overrides.voiceIdstringA voice id that replaces the default voice of the agent.
overrides.systemPromptstringA system prompt that replaces the default prompt of the agent.
overrides.firstMessagestringThe first thing the agent says when the session starts.
overrides.languagestringA language code, for example "en", "es", or "fr".

Client adapter options

elevenlabsRealtime(options?) runs in the browser:

OptionTypeDefaultDescription
connectionMode"websocket" or "webrtc"auto-detectThe transport of the connection.
debugboolean or DebugConfigfalseDebug logs. true turns on all categories. A DebugConfig selects categories or a sink.

Without React

Use RealtimeClient from @tanstack/ai-client:

ts
import { RealtimeClient } from "@tanstack/ai-client";
import { elevenlabsRealtime } from "@tanstack/ai-elevenlabs";

const client = new RealtimeClient({
  getToken: () =>
    fetch("/api/realtime-token", { method: "POST" }).then((r) => r.json()),
  adapter: elevenlabsRealtime(),
  onMessage: (message) => {
    console.log(`${message.role}:`, message.parts);
  },
  onStatusChange: (status) => {
    console.log("Status:", status);
  },
  onModeChange: (mode) => {
    console.log("Mode:", mode);
  },
});

await client.connect();

Audio levels for a visualizer

The hook gives you the audio data of the microphone and the speaker:

ts
import { useRealtimeChat } from "@tanstack/ai-react";
import { elevenlabsRealtime } from "@tanstack/ai-elevenlabs";

export function useVoiceLevels() {
  const {
    inputLevel, // 0 to 1, microphone volume
    outputLevel, // 0 to 1, speaker volume
    getInputFrequencyData, // Uint8Array frequency spectrum
    getOutputFrequencyData,
  } = useRealtimeChat({
    getToken: () =>
      fetch("/api/realtime-token", { method: "POST" }).then((r) => r.json()),
    adapter: elevenlabsRealtime(),
  });

  return { inputLevel, outputLevel, getInputFrequencyData, getOutputFrequencyData };
}

ElevenLabs gives volume levels and frequency data, but no time-domain data. getInputTimeDomainData() and getOutputTimeDomainData() return fixed placeholder arrays. The default sample rate is 16 kHz.

ElevenLabs and OpenAI realtime

ElevenLabsOpenAI
ConfigurationAgent-based. Set voice, personality, and knowledge in the dashboard, or with overrides on the token.Session-based. Set instructions, voice, and more for each session in the useRealtimeChat options.
TokenSigned WebSocket URL, valid for 30 minutesEphemeral API token, valid for about 10 minutes
TransportWebSocket (default) or WebRTCWebRTC
AudioThe @elevenlabs/client SDK records and plays the audioTanStack AI runs the WebRTC connection and the audio tracks
Voice activity detectionElevenLabs does it on its serverserver, semantic, and manual modes
Session changesFixed when the session startsupdateSession() changes the session while it runs
Image inputNot supportedsendImage()
Time-domain dataNot availableAvailable for waveforms

Dialogue and timings

ElevenLabs has a dialogue endpoint that takes up to 10 different voices. Each endpoint also has a version with timestamps. turns and timestamps pick the endpoint for you:

ts
import { generateSpeech } from "@tanstack/ai";
import { elevenlabsSpeech } from "@tanstack/ai-elevenlabs";

const result = await generateSpeech({
  adapter: elevenlabsSpeech("eleven_v4"),
  turns: [
    { text: "Knock knock.", voice: "bYTqZQo3Jz7LQtmGTgwi" },
    { text: "Who is there?", voice: "6lCwbsX1yVjD49QmpkTR" },
  ],
  timestamps: true,
  threadId: "thread-1",
});

// Character timings for the whole clip.
console.log(result.alignment?.unit); // "character"

// One entry per turn, with the voice that spoke it.
for (const segment of result.segments ?? []) {
  console.log(segment.turnIndex, segment.voice, segment.text);
}

Only the dialogue endpoint returns segments. A request with one voice and timestamps: true returns alignment only.

Speech output format

  • wav output is 44.1 kHz, 16-bit mono PCM with a RIFF header.
  • modelOptions.outputFormat overrides format. The adapter then returns the format of ElevenLabs as it is, with no WAV header.

List voices

elevenlabsSpeech reads the voices of your account. This includes the voices that generateVoice() saved:

ts
import { listVoices } from "@tanstack/ai";
import { elevenlabsSpeech } from "@tanstack/ai-elevenlabs";

const { voices } = await listVoices({
  adapter: elevenlabsSpeech("eleven_v4"),
  origins: ["generated", "cloned"],
});

console.log(voices.length);

GET /v1/voices has no origin filter, so the adapter gets all voices and filters them in memory. The famous and high_quality categories both come back as professional.

Behind a proxy

To send each request through a gateway, such as Cloudflare AI Gateway or a company proxy, set baseURL and defaultHeaders. All TanStack AI adapters use these two option names, so one gateway configuration works for all of them.

ts
import { createElevenLabsSpeech } from "@tanstack/ai-elevenlabs";

const adapter = createElevenLabsSpeech("eleven_v4", process.env.ELEVENLABS_API_KEY!, {
  baseURL: "https://gateway.example.com/elevenlabs",
  defaultHeaders: { "cf-aig-authorization": `Bearer ${process.env.GATEWAY_TOKEN}` },
});
  • This works for the speech, audio, transcription, and voice design adapters.
  • baseUrl and headers are aliases. If you set both forms, baseURL and defaultHeaders win.

Limits

  • Realtime does not take images during a conversation.
  • Realtime fixes the session configuration when the connection starts.
  • Realtime needs an agent that you made in the ElevenLabs dashboard.

API reference

FactoryUse it with
elevenlabsRealtimeToken(options)realtimeToken() on the server. Returns a RealtimeTokenAdapter.
elevenlabsRealtime(options?)useRealtimeChat() or RealtimeClient in the browser. Returns a RealtimeAdapter.
elevenlabsSpeech(model, config?) / createElevenLabsSpeech(model, apiKey, config?)generateSpeech() and listVoices()
elevenlabsAudio(model, config?) / createElevenLabsAudio(model, apiKey, config?)generateAudio(), for music and sound effects
elevenlabsTranscription(model, config?) / createElevenLabsTranscription(model, apiKey, config?)generateTranscription()
elevenlabsVoiceDesign(model, config?) / createElevenLabsVoiceDesign(model, apiKey, config?)generateVoice()

The short factories read ELEVENLABS_API_KEY. The create* factories take an explicit API key.

Edit on GitHub
Ollaya
fal.ai