Live mode

Live mode provides real-time transcription for captions, voice agents, contact center applications, and RTMS integrations. Stream audio over WebSocket and receive Captions (partial results) and completed transcript items as speech is detected and processed.

Capabilities#

• Binary PCM16 or Opus audio streaming
• Captions (partial transcription results) before a transcript item is completed
• Recognition in ten languages with zoom-scribe-multilingual and 26 additional locales with whisper
• Speaker diarization
• Custom vocabulary
• Optional Text Polishing
• Returns your source timestamps and metadata, such as participant names, with each transcript item
• Accepts Zoom RTMS audio timestamps and participant details without conversion

Connect#

Open a WebSocket connection to:

wss://api.zoom.us/v2/aiservices/scribe/live

Include the live-asr subprotocol and your Zoom AI Services credentials in the upgrade request. The example sends a token in the Authorization header.

import WebSocket from 'ws';

const socket = new WebSocket(
    'wss://api.zoom.us/v2/aiservices/scribe/live',
    ['live-asr'],
    {
        headers: {
            Authorization: `Bearer ${process.env.ZOOM_ACCESS_TOKEN}`,
        },
    },
);

Configure the session#

After the connection opens, send a session.update event.

{
    "type": "session.update",
    "config": {
        "language": "en-US",
        "model": "zoom-scribe-multilingual",
        "diarization": true,
        "vocabulary": {
            "phrases": ["Project Atlas", "Jane Chen"]
        }
    },
    "audio": {
        "format": "opus"
    },
    "outputs": {
        "text_polishing": {
            "enabled": false
        }
    },
    "reference_id": "live-session-123"
}

Fields#

Use either config.vocabulary or config.vocabulary_id, not both. Send a later session.update to replace an inline vocabulary during the session.

For model and language options, see Speech configuration. For vocabulary options, see Custom vocabulary.

Stream audio#

For PCM16, use signed little-endian, 16 kHz, mono audio and send approximately 100 ms per binary frame. Do not include a WAV header or Base64-encode binary audio.

Opus is supported in Live mode. Set audio.format to opus and send bare Opus packets in binary frames, without a container such as Ogg or WebM. Each packet can contain any valid Opus duration up to 120 ms.

If you don't need to know which participant or source each transcript item came from, send audio as binary frames directly.

Stream audio with metadata#

To attach a timestamp or details about who is speaking, such as a participant ID and name, to audio, send an input_audio_buffer.append JSON text frame immediately followed by exactly one binary audio frame.

{
    "type": "input_audio_buffer.append",
    "timestamp": 1738392033699,
    "metadata": {
        "user_id": 16778240,
        "user_name": "John Smith"
    }
}

timestamp is the source timestamp for the audio. metadata is an application-defined object. Scribe preserves the association and returns the metadata with the corresponding completed transcript item.

Integrate Zoom RTMS#

Forward Zoom RTMS audio events as-is, without changing their participant identifiers:

For merged meeting audio, RTMS uses user_id: 0 and may omit user_name. Other RTMS products can provide another source identifier, such as channel_id; pass it through in metadata.

Handle server events#

{
    "type": "transcription.completed",
    "item_id": "item_123",
    "transcript": "Let's ship this next week.",
    "speaker": "Speaker 1",
    "audio_start_ms": 12420,
    "audio_end_ms": 15180,
    "metadata": {
        "user_id": 16778240,
        "user_name": "John Smith"
    },
    "transcription_latency_ms": 284
}

Enable Captions#

Captions are partial (interim) transcription results returned before a transcript item is completed. Enable them with config.interim_results in session.update.

{
    "type": "session.update",
    "config": {
        "language": "en-US",
        "interim_results": true
    },
    "audio": {
        "format": "pcm16"
    }
}

Partial text is returned in transcription.delta events:

{
    "type": "transcription.delta",
    "item_id": "item_123",
    "delta": "Let's ship"
}

Each delta contains the accumulated partial text for its item_id, not just newly recognized words. Replace the displayed partial text for that item instead of appending each delta.

Use item_id to associate partial results with the corresponding transcription.completed event. When the completed event arrives, replace the partial text with its final transcript, which may correct recognition errors in the partial results.

Captions and Text Polishing are independent. When both are enabled, transcription.delta events return partial results of the original transcript, and each transcription.completed event includes the polished text.

Enable multilingual recognition#

Select zoom-scribe-multilingual to recognize any of its ten supported languages and detect language changes automatically.

{
    "type": "session.update",
    "config": {
        "language": "en-US",
        "model": "zoom-scribe-multilingual"
    },
    "audio": {
        "format": "opus"
    }
}

language is the language Scribe expects at the start of the session. This model detects language changes on its own, so you don't need to set language_detection. Each transcription.completed event includes the detected language, and a language change starts a new transcript item.

Use whisper for languages that zoom-scribe-multilingual doesn't support. The model IDs are the same as in Fast mode. To select Whisper in a Live session, send:

{
    "type": "session.update",
    "config": {
        "language": "en-US",
        "model": "whisper"
    },
    "audio": {
        "format": "pcm16"
    }
}

Enable Text Polishing#

Add outputs.text_polishing to session.update.

{
    "type": "session.update",
    "config": {
        "language": "en-US"
    },
    "audio": {
        "format": "pcm16"
    },
    "outputs": {
        "text_polishing": {
            "enabled": true,
            "include_transcript": true
        }
    }
}

Polished results are returned under outputs.text_polishing.text in transcription.completed events. One polished event can cover several speech items, so don't wait for a transcription.completed event for every item_id. See Text Polishing for response behavior.

Close the session#

1. Stop sending audio.
2. Send { "type": "session.close" }.
3. Continue reading events until the server returns any remaining transcription.completed event and then session.closed.
4. Close the WebSocket.

Session limits#