# Live sessions Voice Agent live sessions use JSON text frames over WebSocket. Input and output audio is Base64-encoded PCM16 inside JSON events. ## Connect Open a WebSocket connection to: **`wss://api.zoom.us/v2/ws/aiservices/voice-agent/live`** Include the `voice-agent.zoom.v2` subprotocol and your [Zoom AI Services credentials](/docs/ai-services/auth/) in the upgrade request. The example sends a token in the `Authorization: Bearer` header. ```javascript import WebSocket from "ws"; const socket = new WebSocket( "wss://api.zoom.us/v2/ws/aiservices/voice-agent/live", ["voice-agent.zoom.v2"], { headers: { Authorization: `Bearer ${process.env.ZOOM_ACCESS_TOKEN}`, }, }, ); ``` ## Configure the session The server sends `session.created` after the connection opens. Your first client event must be `session.update`. Session settings are grouped into three objects: | Object | Configures | | -------- | --------------------------------------------------------------- | | `listen` | Input audio format and speech transcription. | | `think` | The model, instructions, response length, reasoning, and tools. | | `speak` | Output audio format and voice. | Send an empty object to use the defaults for that group: ```json { "type": "session.update", "session": { "listen": {}, "think": {}, "speak": {} } } ``` This example sets each group explicitly: ```json { "type": "session.update", "session": { "listen": { "transcription": { "language": "en", "vocabulary": { "phrases": ["Zoom", "Voice Agent"] } }, "format": { "type": "pcm16", "rate": 16000 } }, "think": { "provider": { "type": "zoom", "model": "gemma-4" }, "instructions": "You are a concise and friendly customer service representative.", "response_token_limit": "inf", "tools": [], "tool_choice": "auto", "reasoning": { "effort": "low" } }, "speak": { "format": { "type": "pcm16" }, "voice": "jenna" } } } ``` The server confirms the effective configuration with `session.updated`. The following example shows the supplied instructions, defaults for omitted settings, and the session `id`: ```json { "type": "session.updated", "session": { "id": "43d2983171fb46af9c6c315e23ae65b9", "listen": { "transcription": { "language": "en" }, "format": { "type": "pcm16", "rate": 16000 } }, "think": { "provider": { "type": "zoom", "model": "gemma-4" }, "instructions": "You are a concise and friendly customer service representative.", "response_token_limit": "inf", "tool_choice": "auto" }, "speak": { "format": { "type": "pcm16", "rate": 16000 }, "voice": "jenna" } } } ``` A session lasts up to 60 minutes. A successful `session.update` returns `session.updated`, which doesn't echo `think.tools`. Keep the tool definitions in your application. ### Listen settings | Field | Default | Description | | ----------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `listen.format.type` | `pcm16` | Input audio encoding. | | `listen.format.rate` | `16000` | Input sample rate in Hz. Any sample rate is supported, such as `8000`, `16000`, `24000`, `44100`, or `48000`. Set it to match the audio you send. | | `listen.transcription.language` | `en` | Language used to transcribe input speech. Only `en` is supported. | | `listen.transcription.vocabulary.phrases` | — | Names and terms to improve transcription accuracy, such as product names. | ### Think settings | Field | Default | Description | | ---------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `think.provider.type` | `zoom` | `zoom` for the built-in model, or `openai` or `anthropic` to use your own provider key. See [Bring your own key](#bring-your-own-key). | | `think.provider.model` | `gemma-4` | Conversational model. With `type` set to `zoom`, use `gemma-4`. | | `think.instructions` | General instructions for spoken conversation | Persistent instructions for the session. | | `think.response_token_limit` | `"inf"` | Maximum tokens generated for each response. A response that reaches the limit ends with status `incomplete`. | | `think.tools` | `[]` | Functions the agent can call. See [Use function calling](#use-function-calling). | | `think.tool_choice` | `auto` | How the agent chooses tools. | | `think.reasoning.effort` | — | Set the reasoning level, such as `low`, in your first `session.update`. | ### Speak settings | Field | Default | Description | | ------------------- | ------- | -------------------------------------------------------------------------- | | `speak.format.type` | `pcm16` | Output audio encoding. | | `speak.format.rate` | `16000` | Output sample rate. Only `16000` is supported. | | `speak.voice` | `jenna` | Voice used for audio responses. See [Available voices](#available-voices). | Turn detection, noise reduction, speech speed, truncation, barge-in, and conversation history trimming use built-in defaults and aren't configurable. ### Bring your own key To use your own OpenAI or Anthropic account, set `think.provider.type` to `openai` or `anthropic`, set `model` to a model from that provider, and pass your provider's API key in `think.provider.headers`. This is your OpenAI or Anthropic key, not your Zoom AI Services credentials: ```json { "type": "session.update", "session": { "listen": {}, "think": { "provider": { "type": "openai", "model": "", "headers": { "X-Api-Key": "" }, "metadata": { "context_window": 1000000, "max_output_tokens": 128000 } }, "instructions": "You are a customer service representative." }, "speak": {} } } ``` `headers` is required when you use your own key. `metadata` describes the model's context window and maximum output tokens. `session.updated` never returns `headers`. ## Stream input audio Send audio continuously, including silence, in `input_audio_buffer.append` events. The audio must match `listen.format`. ```json { "type": "input_audio_buffer.append", "audio": "" } ``` ### Start another response Only one response can be active at a time. Before sending `response.create` for another response, wait for `response.done` for the active response. If you send `response.create` while a response is still active, the server returns a nonfatal error indicating that there is an active response. Wait for `response.done`, then retry. Receiving `response.output_audio.done` or `response.function_call_arguments.done` does not mean the response has ended. Use `response.done` to determine when another response can start. ## Handle server events Voice Agent emits events as input is detected and responses are generated. | Event | Description | | ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | `session.created` | Connection established with the default session configuration. | | `session.updated` | A `session.update` was applied. | | `input_audio_buffer.speech_started` | Speech was detected in the input stream. | | `input_audio_buffer.speech_stopped` | The detected speech turn ended. | | `conversation.item.input_audio_transcription.completed` | The user's input transcript is ready. | | `response.created` | An agent response started. | | `response.output_item.added` | A response output item was added, including its `item.id`. | | `response.output_audio.delta` | A Base64-encoded chunk of agent speech. | | `response.output_audio.done` | All audio for the response output has been sent. | | `response.output_audio_transcript.delta` | An incremental transcript of agent speech. | | `response.function_call_arguments.done` | A function call and its complete arguments are ready. | | `response.done` | The response ended with status `completed`, `cancelled`, `failed`, or `incomplete` (token limit reached). | | `error` | A request validation or service error occurred. | Server events are wrapped in an envelope with a `type` and a `payload` object. `payload` contains the complete event, including its `type`, so read the event from `payload`. The event examples on this page show the event inside `payload`. ```javascript socket.on("message", (data) => { const envelope = JSON.parse(data.toString()); const event = envelope.payload ?? envelope; if (event.type === "response.output_audio.delta") { playPcm16(Buffer.from(event.delta, "base64")); } if (event.type === "error") { console.error(event.error); } }); ``` ## Cancel or truncate a response Use the server-generated IDs from response events: - `response.created.response.id` identifies the response. Pass it as `response_id` in `response.cancel`. - `response.output_item.added.item.id` identifies an output item. Audio events also include `item_id`; use the ID of the item whose audio you are truncating in `conversation.item.truncate`. When a user interrupts audio that is already playing: 1. Send `response.cancel` to stop generation. 2. Stop local playback and record the last audio offset the user heard. 3. Send `conversation.item.truncate` so the conversation history matches the played audio. ```json { "type": "response.cancel", "response_id": "resp_abc123" } ``` ```json { "type": "conversation.item.truncate", "item_id": "item_abc123", "content_index": 0, "audio_end_ms": 1500 } ``` ## Use function calling Add function definitions to `think.tools`, and set `think.tool_choice` to `auto` (the model decides), `none`, or `required` (call at least one tool). ```json { "type": "session.update", "session": { "listen": {}, "think": { "tools": [ { "type": "function", "name": "get_order_status", "description": "Get the current status of an order.", "parameters": { "type": "object", "properties": { "order_id": { "type": "string" } }, "required": ["order_id"] } } ], "tool_choice": "auto" }, "speak": {} } } ``` The server sends `response.function_call_arguments.done` when the function's arguments are complete: ```json { "type": "response.function_call_arguments.done", "event_id": "event_abc123", "response_id": "resp_abc123", "item_id": "item_abc123", "output_index": 0, "call_id": "call_abc123", "name": "get_order_status", "arguments": "{\"order_id\":\"123\"}" } ``` Parse `arguments` as JSON and execute the named function in your application. Return its result as a `function_call_output` item using the event's `call_id`. The `call_id` links the result to the function call; it is not the response or item ID. ```json { "type": "conversation.item.create", "item": { "type": "function_call_output", "call_id": "call_abc123", "output": "{\"status\":\"shipped\",\"estimated_delivery\":\"Friday\"}" } } ``` Wait for `response.done` for the response that requested the function before sending `response.create` to request the agent's next response. ```json { "type": "response.create" } ``` ## Available voices Set `speak.voice` to one of these ten voice IDs. The default is `jenna`. | Voice | Voice ID | Accent | Character | | ------ | -------- | ---------------- | --------------------------- | | Alex | `alex` | American English | Charming and friendly | | Ariel | `ariel` | American English | Professional and clear | | Ava | `ava` | American English | Calm and empathetic | | Brooke | `brooke` | American English | Calm and professional | | Hazel | `hazel` | American English | Enthusiastic and warm | | Jenna | `jenna` | American English | Clear and smooth | | Mark | `mark` | American English | Professional and empathetic | | Reid | `reid` | American English | Calm and crisp | | Scott | `scott` | American English | Friendly and deep | | Tony | `tony` | British English | Charming and eloquent | ## Handle errors The `error.type` field is `invalid_request_error` for recoverable client errors or `server_error` for service failures. The `error.message` field describes the failure. ```json { "type": "error", "event_id": "event_abc123", "error": { "type": "invalid_request_error", "message": "" } } ``` When configuration validation fails, the server sends an error event. Unknown voices and unsupported input formats produce validation errors. Check `session.updated` for the effective settings. A rejected `session.update` does not disconnect the client. Correct the configuration and send `session.update` again on the same connection; a successful update returns `session.updated`. An error indicating that there is an active response is also nonfatal. Wait for that response's `response.done` before retrying `response.create`.