Live mode

Live mode provides real-time speech translation for meetings, live events, classrooms, and customer support. Stream audio over WebSocket and receive translated text and speech as the speaker talks.

Capabilities#

• Base64-encoded PCM16 audio streaming in JSON events
• Translated text and synthesized speech in the target language
• Optional source-language transcription
• Final text and audio results that are never revised
• One source language, target language, and voice per connection

Connect#

Open a WebSocket connection to:

wss://api.zoom.us/v2/ws/aiservices/interpreter/live

Include the interpreter.zoom.v1 subprotocol and your Zoom AI Services credentials in the upgrade request. The subprotocol is required. The example sends a token in the Authorization header.

import WebSocket from 'ws';

const socket = new WebSocket(
    'wss://api.zoom.us/v2/ws/aiservices/interpreter/live',
    ['interpreter.zoom.v1'],
    {
        headers: {
            Authorization: `Bearer ${process.env.ZOOM_ACCESS_TOKEN}`,
        },
    },
);

Set ZOOM_ACCESS_TOKEN to a JWT generated from your API credentials. To use an API key instead, send it in the x-api-key header.

The following failures reject the connection before the WebSocket opens, so handle connection errors as well as error events.

Reconnecting starts a new session. You can't resume a previous session.

Configure the session#

When the connection opens, the server sends session.created. It includes session.id, session.type (translation), session.expires_at (seconds since the Unix epoch), and the default session.audio settings. The languages and voice are null until you set them, and source transcription is off.

Send session.update as your first event. You don't need to wait for session.created.

{
    "type": "session.update",
    "event_id": "client-evt-00",
    "session": {
        "audio": {
            "input": {
                "language": "en-US",
                "transcription": {}
            },
            "output": {
                "language": "es-ES",
                "voice": "sofia"
            }
        }
    }
}

For supported languages and voices, see Languages and voices. Voice IDs and language tags are case-insensitive; session.updated returns them normalized (voices in lowercase).

Wait for session.updated before you send audio or session.close. Sending either one before session.update ends the session with a fatal invalid_message error.

If the configuration is rejected, the server sends an error event, and error.param names the field to fix. When error.fatal is false, correct the field and send session.update again. Repeated rejections can end the session, so always check error.fatal.

Accepted configuration#

{
    "type": "session.updated",
    "event_id": "event_0002",
    "session": {
        "id": "st-0123456789abcdef01234567",
        "type": "translation",
        "expires_at": 1793001600,
        "audio": {
            "input": {
                "language": "en-US",
                "transcription": {}
            },
            "output": {
                "language": "es-ES",
                "voice": "sofia"
            }
        }
    }
}

session.updated confirms the configuration and returns the language tags in normalized form. The server sends it once. Keep session.id for support requests.

You can't change the configuration after it's accepted. A later session.update returns a non-fatal invalid_request error with param: session, and the session continues unchanged. To change languages or voice, open a new connection.

Output selection#

The service always produces translated text and translated audio. The transcription setting controls only whether you also receive source-language text.

For source transcription without translation, use Scribe Live mode.

Send audio#

After session.updated, send each audio chunk as a JSON event. Base64-encode raw signed 16-bit little-endian PCM, 16,000 Hz, mono, without a WAV header. The service doesn't resample input, so convert audio to this format before sending it.

{
    "type": "session.input_audio_buffer.append",
    "event_id": "client-evt-01",
    "audio": "Base64EncodedAudioData"
}

Send 100–200 ms of audio per event (3,200–6,400 bytes before Base64 encoding). Each event can carry at most 500 ms of audio (16,000 bytes). A larger event ends the session with a fatal invalid_message error. A WebSocket message larger than 1 MiB closes the connection with close code 1009.

Send audio continuously as it becomes available, including silence and pauses. The server doesn't acknowledge audio events, and you don't need to wait for output before sending the next chunk.

Handle server events#

The server sends JSON events. Each includes a type and an event_id. Ignore fields you don't recognize, so that your client keeps working when new fields are added.

Text deltas#

{
    "type": "session.input_transcript.delta",
    "event_id": "event_0121",
    "delta": " I can hear you."
}

{
    "type": "session.output_transcript.delta",
    "event_id": "event_0122",
    "delta": " Te escucho."
}

Each text delta is final and is never revised. Append the fragments for each transcript in the order you receive them. Deltas already include any spaces needed between them, so don't add your own.

Audio deltas#

{
    "type": "session.output_audio.delta",
    "event_id": "event_0123",
    "delta": "Base64EncodedAudioDelta",
    "format": "pcm_s16le",
    "sample_rate": 16000,
    "channels": 1
}

Each audio delta is final and is never revised. Decode delta from Base64 and process the PCM bytes in the order you receive them. Output audio is always signed 16-bit little-endian PCM, 16 kHz, mono, without a header.

Events from the source transcript, translated transcript, and translated audio arrive interleaved. Each stream is in order, but the streams aren't synchronized with each other, and text fragments don't map one-to-one to audio chunks.

Don't block the receive handler while you process output. Buffer output in your application; the server doesn't slow down for slow clients.

JavaScript event handling#

The following extends the connection example above. Your application calls sendPcm for each input PCM chunk after configuration and finishInput once at the end. handleOutputAudio is an application function that processes the translated PCM without blocking the receive handler.

let configured = false;
let inputEnded = false;
let sourceTranscript = '';
let translatedTranscript = '';
let completed = false;

socket.on('open', () => {
    socket.send(JSON.stringify({
        type: 'session.update',
        session: {
            audio: {
                input: { language: 'en-US', transcription: {} },
                output: { language: 'es-ES', voice: 'sofia' },
            },
        },
    }));
});

function sendPcm(pcmBuffer) {
    if (!configured || inputEnded || socket.readyState !== WebSocket.OPEN) {
        throw new Error('The session is not accepting audio');
    }
    if (!pcmBuffer.length || pcmBuffer.length % 2 || pcmBuffer.length > 16000) {
        throw new Error('Send non-empty, complete PCM16 samples within the size limit');
    }
    socket.send(JSON.stringify({
        type: 'session.input_audio_buffer.append',
        audio: pcmBuffer.toString('base64'),
    }));
}

function finishInput() {
    if (!configured || inputEnded || socket.readyState !== WebSocket.OPEN) {
        throw new Error('The session is not ready to end input');
    }
    inputEnded = true;
    socket.send(JSON.stringify({ type: 'session.close' }));
}

socket.on('message', (data) => {
    const event = JSON.parse(data.toString());
    switch (event.type) {
        case 'session.created':
            console.log('Session:', event.session.id);
            break;
        case 'session.updated':
            configured = true;
            // Your application can now call sendPcm for each input chunk.
            break;
        case 'session.input_transcript.delta':
            sourceTranscript += event.delta;
            break;
        case 'session.output_transcript.delta':
            translatedTranscript += event.delta;
            break;
        case 'session.output_audio.delta':
            handleOutputAudio(Buffer.from(event.delta, 'base64'));
            break;
        case 'error':
            console.error(event.error);
            if (event.error.fatal) {
                configured = false;
                socket.close();
            }
            // For a non-fatal configuration error, correct session.update and resend.
            break;
        case 'session.closed':
            completed = event.reason === 'completed';
            configured = false;
            console.log('Session ended:', event.reason);
            socket.close();
            break;
    }
});

socket.on('error', (error) => {
    console.error('WebSocket failure:', error.message);
});

socket.on('close', () => {
    configured = false;
    if (!completed) {
        console.error('Translation ended without completed confirmation');
    }
});

In production, also set a timeout while waiting for session.closed, and surface non-fatal configuration errors to your users or retry logic. Keep the same message handler running after you end input.

Close the session#

1. Stop sending audio.
2. Send { "type": "session.close" } once.
3. Continue reading events until the server returns any remaining text and audio and then session.closed.
4. Close the WebSocket if the server hasn't already closed it.

If you close the socket before session.closed arrives, you lose any output still in progress.

The session.closed event includes a reason:

{
    "type": "session.closed",
    "event_id": "event_0999",
    "reason": "completed"
}

completed means all output was delivered. Any other reason means the server ended the session early. session.closed is always the last event, and the server then closes the WebSocket. Fatal errors and network failures can close the connection without session.closed, so a normal WebSocket close code alone doesn't mean translation completed.

Session limits#

Your account can have lower limits. session.expires_at gives the time when the session reaches its maximum duration.

The idle timer starts when the connection opens and resets with every event you send. Server output doesn't reset it, and it stops after you send session.close. Keep sending audio, including silence, while input is active.

Handle session_expired, session_limit, idle_timeout, and unexpected connection closure. To continue translating, start a new session.

Handle errors#

{
    "type": "error",
    "event_id": "event_0890",
    "error": {
        "type": "invalid_request_error",
        "code": "invalid_request",
        "message": "voice is not available for the target language",
        "param": "session.audio.output.voice",
        "event_id": "client-evt-00",
        "retryable": false,
        "fatal": false
    }
}

The top-level event_id identifies the error event. error.event_id is the event_id of your event that caused the error, or null if you didn't set one. error.param names the field that caused the error, or is null. Use error.code in your logic; error.message is for people and can change.

error.type is invalid_request_error for problems you can fix, or server_error for service failures. Only errors for session.update can be non-fatal. When error.fatal is true, the server closes the connection and sends no more events.

retryable: true means a new connection may succeed. Resending audio from the failed session can duplicate translated output. For support requests, keep the session ID, configuration, and errors. Never log credentials.