# Live mode Live mode provides real-time speech translation for meetings, live events, classrooms, and customer support. Stream audio over WebSocket and receive translated text and speech as the speaker talks. ## Capabilities - Base64-encoded PCM16 audio streaming in JSON events - Translated text and synthesized speech in the target language - Optional source-language transcription - Final text and audio results that are never revised - One source language, target language, and voice per connection ## Connect Open a WebSocket connection to: **`wss://api.zoom.us/v2/ws/aiservices/interpreter/live`** Include the `interpreter.zoom.v1` subprotocol and your [Zoom AI Services credentials](/docs/ai-services/auth/) in the upgrade request. The subprotocol is required. The example sends a token in the `Authorization` header. ```javascript import WebSocket from "ws"; const socket = new WebSocket( "wss://api.zoom.us/v2/ws/aiservices/interpreter/live", ["interpreter.zoom.v1"], { headers: { Authorization: `Bearer ${process.env.ZOOM_ACCESS_TOKEN}`, }, }, ); ``` > Create the WebSocket connection from a trusted backend. Browser WebSocket clients cannot set the `Authorization` header, and placing credentials in client-side code exposes them. Set `ZOOM_ACCESS_TOKEN` to a JWT generated from your API credentials. To use an API key instead, send it in the `x-api-key` header. The following failures reject the connection before the WebSocket opens, so handle connection errors as well as `error` events. | Failure | Handling | | ------------------------------------ | -------------------------------------------------------------------------------------- | | `401` invalid or missing credentials | Check your API key or JWT. | | `402 QUOTA_EXCEEDED` | Your API key reached its credit limit. Raise the limit or wait until the limit resets. | | `503 SERVICE_DISABLED` | The Interpreter API isn't enabled for your account. | | `503 CAPACITY_EXCEEDED` | Too many concurrent sessions. Retry with exponential backoff and jitter. | Reconnecting starts a new session. You can't resume a previous session. ## Configure the session When the connection opens, the server sends `session.created`. It includes `session.id`, `session.type` (`translation`), `session.expires_at` (seconds since the Unix epoch), and the default `session.audio` settings. The languages and voice are `null` until you set them, and source transcription is off. Send `session.update` as your first event. You don't need to wait for `session.created`. ```json { "type": "session.update", "event_id": "client-evt-00", "session": { "audio": { "input": { "language": "en-US", "transcription": {} }, "output": { "language": "es-ES", "voice": "sofia" } } } } ``` For supported languages and voices, see [Languages and voices](/docs/ai-services/interpreter/#languages-and-voices). Voice IDs and language tags are case-insensitive; `session.updated` returns them normalized (voices in lowercase). | Field | Type | Required | Description | | ----------------------------------- | -------------- | -------- | -------------------------------------------------------------------------------------------------------------------------- | | `type` | string | Yes | Set to `session.update`. | | `event_id` | string | No | Client-generated event identifier, returned as `error.event_id` if the event is rejected. | | `session` | object | Yes | Session configuration. | | `session.audio.input.language` | string | Yes | Source BCP-47 language tag; no default. | | `session.audio.input.transcription` | object or null | No | Set to `{}` to receive source transcript events. Omit it or set it to `null` to turn them off. The object takes no fields. | | `session.audio.output.language` | string | Yes | Target BCP-47 language tag; must differ from the source. No default. | | `session.audio.output.voice` | string | Yes | Voice that supports the target language; no default. | Wait for `session.updated` before you send audio or `session.close`. Sending either one before `session.update` ends the session with a fatal `invalid_message` error. If the configuration is rejected, the server sends an `error` event, and `error.param` names the field to fix. When `error.fatal` is `false`, correct the field and send `session.update` again. Repeated rejections can end the session, so always check `error.fatal`. ### Accepted configuration ```json { "type": "session.updated", "event_id": "event_0002", "session": { "id": "st-0123456789abcdef01234567", "type": "translation", "expires_at": 1793001600, "audio": { "input": { "language": "en-US", "transcription": {} }, "output": { "language": "es-ES", "voice": "sofia" } } } } ``` `session.updated` confirms the configuration and returns the language tags in normalized form. The server sends it once. Keep `session.id` for support requests. You can't change the configuration after it's accepted. A later `session.update` returns a non-fatal `invalid_request` error with `param: session`, and the session continues unchanged. To change languages or voice, open a new connection. ### Output selection The service always produces translated text and translated audio. The `transcription` setting controls only whether you also receive source-language text. | Outputs your application uses | Configuration and handling | | -------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | Source text, translated text, and translated audio | Set `session.audio.input.transcription` to `{}` and handle all three delta events. | | Translated text and audio | Omit source transcription and handle both output delta events. | | Translated audio only | Omit source transcription and ignore translated-text events. | | Translated text only | Ignore translated-audio events. A voice is still required, and translated audio is still produced. | For source transcription without translation, use [Scribe Live mode](/docs/ai-services/scribe/live-mode/). ## Send audio After `session.updated`, send each audio chunk as a JSON event. Base64-encode raw signed 16-bit little-endian PCM, 16,000 Hz, mono, without a WAV header. The service doesn't resample input, so convert audio to this format before sending it. ```json { "type": "session.input_audio_buffer.append", "event_id": "client-evt-01", "audio": "Base64EncodedAudioData" } ``` | Field | Type | Required | Description | | ---------- | ------ | -------- | ----------------------------------------------------------------------------------------------------------- | | `type` | string | Yes | Set to `session.input_audio_buffer.append`. | | `event_id` | string | No | Client-generated identifier for error correlation. | | `audio` | string | Yes | Non-empty Base64-encoded PCM chunk of complete 16-bit samples, up to 500 ms (16,000 bytes) before encoding. | Send 100–200 ms of audio per event (3,200–6,400 bytes before Base64 encoding). Each event can carry at most 500 ms of audio (16,000 bytes). A larger event ends the session with a fatal `invalid_message` error. A WebSocket message larger than 1 MiB closes the connection with close code `1009`. Send audio continuously as it becomes available, including silence and pauses. The server doesn't acknowledge audio events, and you don't need to wait for output before sending the next chunk. ## Handle server events The server sends JSON events. Each includes a `type` and an `event_id`. Ignore fields you don't recognize, so that your client keeps working when new fields are added. | Event | Description | | --------------------------------- | --------------------------------------------------------------------- | | `session.created` | First server event; reports the session ID and default configuration. | | `session.updated` | Confirms the accepted configuration; audio input may begin. | | `session.input_transcript.delta` | Final source-text fragment, if source transcription is enabled. | | `session.output_transcript.delta` | Final translated-text fragment. | | `session.output_audio.delta` | Base64-encoded translated PCM audio chunk. | | `session.closed` | Last server event; inspect the termination reason. | | `error` | Reports an error and whether the session can continue. | ### Text deltas ```json { "type": "session.input_transcript.delta", "event_id": "event_0121", "delta": " I can hear you." } ``` ```json { "type": "session.output_transcript.delta", "event_id": "event_0122", "delta": " Te escucho." } ``` Each text delta is final and is never revised. Append the fragments for each transcript in the order you receive them. Deltas already include any spaces needed between them, so don't add your own. ### Audio deltas ```json { "type": "session.output_audio.delta", "event_id": "event_0123", "delta": "Base64EncodedAudioDelta", "format": "pcm_s16le", "sample_rate": 16000, "channels": 1 } ``` Each audio delta is final and is never revised. Decode `delta` from Base64 and process the PCM bytes in the order you receive them. Output audio is always signed 16-bit little-endian PCM, 16 kHz, mono, without a header. Events from the source transcript, translated transcript, and translated audio arrive interleaved. Each stream is in order, but the streams aren't synchronized with each other, and text fragments don't map one-to-one to audio chunks. Don't block the receive handler while you process output. Buffer output in your application; the server doesn't slow down for slow clients. ### JavaScript event handling The following extends the connection example above. Your application calls `sendPcm` for each input PCM chunk after configuration and `finishInput` once at the end. `handleOutputAudio` is an application function that processes the translated PCM without blocking the receive handler. ```javascript let configured = false; let inputEnded = false; let sourceTranscript = ""; let translatedTranscript = ""; let completed = false; socket.on("open", () => { socket.send( JSON.stringify({ type: "session.update", session: { audio: { input: { language: "en-US", transcription: {} }, output: { language: "es-ES", voice: "sofia" }, }, }, }), ); }); function sendPcm(pcmBuffer) { if (!configured || inputEnded || socket.readyState !== WebSocket.OPEN) { throw new Error("The session is not accepting audio"); } if (!pcmBuffer.length || pcmBuffer.length % 2 || pcmBuffer.length > 16000) { throw new Error( "Send non-empty, complete PCM16 samples within the size limit", ); } socket.send( JSON.stringify({ type: "session.input_audio_buffer.append", audio: pcmBuffer.toString("base64"), }), ); } function finishInput() { if (!configured || inputEnded || socket.readyState !== WebSocket.OPEN) { throw new Error("The session is not ready to end input"); } inputEnded = true; socket.send(JSON.stringify({ type: "session.close" })); } socket.on("message", (data) => { const event = JSON.parse(data.toString()); switch (event.type) { case "session.created": console.log("Session:", event.session.id); break; case "session.updated": configured = true; // Your application can now call sendPcm for each input chunk. break; case "session.input_transcript.delta": sourceTranscript += event.delta; break; case "session.output_transcript.delta": translatedTranscript += event.delta; break; case "session.output_audio.delta": handleOutputAudio(Buffer.from(event.delta, "base64")); break; case "error": console.error(event.error); if (event.error.fatal) { configured = false; socket.close(); } // For a non-fatal configuration error, correct session.update and resend. break; case "session.closed": completed = event.reason === "completed"; configured = false; console.log("Session ended:", event.reason); socket.close(); break; } }); socket.on("error", (error) => { console.error("WebSocket failure:", error.message); }); socket.on("close", () => { configured = false; if (!completed) { console.error("Translation ended without completed confirmation"); } }); ``` In production, also set a timeout while waiting for `session.closed`, and surface non-fatal configuration errors to your users or retry logic. Keep the same message handler running after you end input. ## Close the session 1. Stop sending audio. 2. Send `{ "type": "session.close" }` once. 3. Continue reading events until the server returns any remaining text and audio and then `session.closed`. 4. Close the WebSocket if the server hasn't already closed it. If you close the socket before `session.closed` arrives, you lose any output still in progress. The `session.closed` event includes a `reason`: ```json { "type": "session.closed", "event_id": "event_0999", "reason": "completed" } ``` `completed` means all output was delivered. Any other `reason` means the server ended the session early. `session.closed` is always the last event, and the server then closes the WebSocket. Fatal errors and network failures can close the connection without `session.closed`, so a normal WebSocket close code alone doesn't mean translation completed. ## Session limits | Limit | Value | | ------------------------ | -------------------------------- | | Maximum session duration | 120 minutes | | Idle timeout | 30 seconds without client events | Your account can have lower limits. `session.expires_at` gives the time when the session reaches its maximum duration. The idle timer starts when the connection opens and resets with every event you send. Server output doesn't reset it, and it stops after you send `session.close`. Keep sending audio, including silence, while input is active. Handle `session_expired`, `session_limit`, `idle_timeout`, and unexpected connection closure. To continue translating, start a new session. ## Handle errors ```json { "type": "error", "event_id": "event_0890", "error": { "type": "invalid_request_error", "code": "invalid_request", "message": "voice is not available for the target language", "param": "session.audio.output.voice", "event_id": "client-evt-00", "retryable": false, "fatal": false } } ``` The top-level `event_id` identifies the error event. `error.event_id` is the `event_id` of your event that caused the error, or `null` if you didn't set one. `error.param` names the field that caused the error, or is `null`. Use `error.code` in your logic; `error.message` is for people and can change. `error.type` is `invalid_request_error` for problems you can fix, or `server_error` for service failures. Only errors for `session.update` can be non-fatal. When `error.fatal` is `true`, the server closes the connection and sends no more events. | Code | Meaning and handling | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `invalid_message` | Malformed JSON, an unknown field or event type, invalid audio, or events sent in the wrong order. For a non-fatal `session.update` error, correct and resend it. Otherwise, fix your client before reconnecting. | | `invalid_request` | The service doesn't support the language, voice, or language pair. Correct the field in `error.param` and send `session.update` again. | | `policy_violation` | The setting is supported, but your account doesn't permit it. Use a setting your account permits, or contact support to enable it. | | `provider_unhealthy` | The translation service failed. If `retryable` is `true`, reconnect with backoff. | | `provider_timeout` | The translation service timed out. If `retryable` is `true`, reconnect with backoff. Contact support if it keeps happening. | | `session_expired` | The session reached its maximum duration. Start a new session if needed. | | `session_limit` | The session reached a duration limit for your account. Start a new session if needed. | | `idle_timeout` | The server received no event from your client within the idle timeout (30 seconds by default). Start a new session when you have audio to send. | | `auth_unavailable` | The service couldn't authenticate with an internal dependency. If `retryable` is `true`, reconnect with backoff. Contact support if it keeps happening. | | `internal_error` | Unexpected service failure. Reconnect with backoff if `retryable` is `true`; otherwise contact support. | `retryable: true` means a new connection may succeed. Resending audio from the failed session can duplicate translated output. For support requests, keep the session ID, configuration, and errors. Never log credentials.