Live sessions

Voice Agent live sessions use JSON text frames over WebSocket. Input and output audio is Base64-encoded PCM16 inside JSON events.

Connect#

Open a WebSocket connection to:

wss://api.zoom.us/v2/aiservices/voice-agent/live

Include the voice-agent.zoom.v1 subprotocol and a Zoom AI Services JWT in the Authorization: Bearer header of the upgrade request.

import WebSocket from 'ws';

const socket = new WebSocket(
  'wss://api.zoom.us/v2/aiservices/voice-agent/live',
  ['voice-agent.zoom.v1'],
  {
    headers: {
      Authorization: `Bearer ${process.env.ZOOM_ACCESS_TOKEN}`,
    },
);

Configure the session#

The server sends session.created after the connection opens. Your first client event must be session.update.

{
    "type": "session.update",
    "session": {
        "type": "realtime",
        "model": "gemma-4",
        "output_modalities": ["audio"],
        "instructions": "You are a concise and friendly support agent.",
        "audio": {
            "input": {
                "format": {
                    "type": "audio/pcm",
                    "rate": 16000
                },
                "transcription": {
                    "model": "zoom_speech",
                    "language": "en"
                },
                "turn_detection": {
                    "type": "server_vad",
                    "threshold": 0.5,
                    "prefix_padding_ms": 300,
                    "silence_duration_ms": 500,
                    "create_response": true,
                    "interrupt_response": true
                }
            },
            "output": {
                "format": {
                    "type": "audio/pcm",
                    "rate": 16000
                },
                "voice": "jenna",
                "speed": 1.0
            }
        },
        "max_output_tokens": "inf"
    }
}

The server confirms the effective configuration with session.updated.

Core settings#

Audio formats#

Set audio.output.format.rate to 16000. Other output sample rates are not supported.

Turn detection#

Set audio.input.turn_detection.type to server_vad or semantic_vad, or set audio.input.turn_detection to null:

For automatic modes, create_response controls whether the API starts an agent response after each turn. interrupt_response controls whether new user speech interrupts a response in progress.

For server_vad, configure these fields in audio.input.turn_detection:

semantic_vad accepts all of the server_vad fields above, plus:

Conversation history#

Set truncation to control how history is trimmed as it approaches the model's token limit. Truncation removes the oldest messages first.

Stream input audio#

Send audio continuously, including silence, in input_audio_buffer.append events.

{
    "type": "input_audio_buffer.append",
    "audio": "<base64-encoded audio>"
}

In manual turn mode, end a user turn by sending these events in order. If another response is active, wait for its response.done before sending response.create:

{
    "type": "input_audio_buffer.commit"
}

{
    "type": "response.create"
}

Use input_audio_buffer.clear to discard audio that has not been committed.

Start another response#

Only one response can be active at a time. Before sending response.create for another response, wait for response.done for the active response. If you send response.create while a response is still active, the server returns a nonfatal error indicating that there is an active response. Wait for response.done, then retry.

Receiving response.output_audio.done or response.function_call_arguments.done does not mean the response has ended. Use response.done to determine when another response can start.

Handle server events#

Voice Agent emits events as input is detected and responses are generated.

Server events are wrapped in an envelope with a type and a payload object. payload contains the complete event, including its type, so read the event from payload. The event examples on this page show the event inside payload.

socket.on('message', (data) => {
    const envelope = JSON.parse(data.toString());
    const event = envelope.payload ?? envelope;

    if (event.type === 'response.output_audio.delta') {
        playPcm16(Buffer.from(event.delta, 'base64'));
    }

    if (event.type === 'error') {
        console.error(event.error);
    }
});

Cancel or truncate a response#

Use the server-generated IDs from response events:

• response.created.response.id identifies the response. Pass it as response_id in response.cancel.
• response.output_item.added.item.id identifies an output item. Audio events also include item_id; use the ID of the item whose audio you are truncating in conversation.item.truncate.

When a user interrupts audio that is already playing:

1. Send response.cancel to stop generation.
2. Stop local playback and record the last audio offset the user heard.
3. Send conversation.item.truncate so the conversation history matches the played audio.

{
    "type": "response.cancel",
    "response_id": "resp_abc123"
}

{
    "type": "conversation.item.truncate",
    "item_id": "item_abc123",
    "content_index": 0,
    "audio_end_ms": 1500
}

Use function calling#

Add function definitions to session.tools and set tool_choice to auto (the model decides), none, required (call at least one tool), or a specific function, such as { "type": "function", "name": "get_order_status" }.

{
    "type": "session.update",
    "session": {
        "tools": [
            {
                "type": "function",
                "name": "get_order_status",
                "description": "Get the current status of an order.",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "order_id": {
                            "type": "string"
                        }
                    },
                    "required": ["order_id"]
                }
            }
        ],
        "tool_choice": "auto"
    }
}

The server sends response.function_call_arguments.done when the function's arguments are complete:

{
    "type": "response.function_call_arguments.done",
    "event_id": "event_abc123",
    "response_id": "resp_abc123",
    "item_id": "item_abc123",
    "output_index": 0,
    "call_id": "call_abc123",
    "name": "get_order_status",
    "arguments": "{\"order_id\":\"123\"}"
}

Parse arguments as JSON and execute the named function in your application. Return its result as a function_call_output item using the event's call_id. The call_id links the result to the function call; it is not the response or item ID.

{
    "type": "conversation.item.create",
    "item": {
        "type": "function_call_output",
        "call_id": "call_abc123",
        "output": "{\"status\":\"shipped\",\"estimated_delivery\":\"Friday\"}"
    }
}

Wait for response.done for the response that requested the function before sending response.create to request the agent's next response.

{
    "type": "response.create"
}

Available voices#

Set audio.output.voice to one of these ten voice IDs. The default is jenna.

Handle errors#

The error.type field is invalid_request_error for recoverable client errors or server_error for service failures. The error.message field describes the failure.

{
    "type": "error",
    "event_id": "event_abc123",
    "error": {
        "type": "invalid_request_error",
        "message": "invalid client event: session.audio.output.format.rate: Input should be 16000"
    }
}

Invalid settings, unknown voices, unsupported formats, and missing fields produce an error event; correct the request before retrying it. A rejected session.update does not disconnect the client. Correct the configuration and send session.update again on the same connection; a successful update returns session.updated.

An error indicating that there is an active response is also nonfatal. Wait for that response's response.done before retrying response.create.