Handling media data

Realtime Media Streams (RTMS) delivers audio and transcript data from Zoom Video SDK sessions over WebSocket connections between your application and the RTMS servers.

After establishing a connection to the RTMS server, your application receives media data based on:

• The formats specified in your media handshake request
• The availability of data in the session

Each media data type has specific formats and best practices for processing the data efficiently.

Audio

Audio data is available per participant and as a merged packet of all participants.

By default, audio data is sent as uncompressed raw PCM (L16) data with a 16kHz sample rate and mono channels.

The RTMS server sends audio data packets from participants in base64-encoded binary format. Each packet of data contains the user's participant identifier (user_id), username (user_name), and the timestamp (timestamp) of the audio data.

Example audio packet:

{
    "msg_type": 14,
    "content": {
        "user_id": 16778240,
        "user_name": "John Smith",
        "data": "Hw1kDacNAA4sDkMOAQ5eDekMgAzXCw0L4Al4CDwHBAayBEwDmwHD/wn+rfyQ+3z6Z/k4+Ef3jvb09aj1YPUF9QL1OfVt9eH1f/YE96P3jPib+dX6Pvyr/ef+IABcAZ4CmQN3BFAF6QVxBs0G9wbjBqYGXwYFBm8FxwT/Aw4DHAIkARYA7v6o/T382/qq+YD4fPd99mz1p/Qq9KTzcfNv807zcPPQ8z70yfQ89az1SPbO9m/3Lfjf+I75M/rz+pT7KPzs/In9Fv7G/lP/q/8GAG4AtgBHAcABDQKtAkwDIgRQBYQGrgfyCEgKigvDDP4NFw/8D78QhBErEqoS/BL5EsYSfRLnEXARwhCMD1kO8ww8C5gJ5gfrBe8D3AGp/7P9IPyY+hL5t/dl9mv10fQ89NjzvvO78/HzbfTb9F71I/bw9vL3Kvlh+pX71/wr/m//owDCAbQCkwNoBPYEYQWxBZ0FZQUdBYkEBgRkA64CAgIdAT0AZf+T/tD93vz9+xf7B/o8+WX4pvcE90L2n/VM9d70pPSe9IL0nfTa9BX1cfXK9QP2S/a69hH3cPfx91P4w/hO+cr5XPr2+l/71/t9/AP9f/0R/p/+Pv/5/7IAWAH+AeMCAwQ7BaAG+wcmCZEKKQy8DT4PWhBXEVYSVBOgFM0VMxZSFhYW4BXTFUMVNBSzEtIQJg99Da0LogkXB40ESgJCAHj+ivyA+qn4KfcH9hr1JvRf89byq/Lt8kjzhvPy85P0n/X/9kn4lvni+kj84P1S/4wAuwHUAvcD9wS/BUwGaAZcBmcGTQb5BUkFQgQzAzkCVQEgALX+U/3N+4L6XfkU+Nj2o/WB9MTzOfO38g==",
        "timestamp": 1738392033699
    }
}

Send rate

The default interval is 20ms between audio packets. You can configure this in multiples of 20ms, up to a maximum of 1000ms. If you specify an interval above 1000ms in handshake requests, RTMS will change it to 1000ms.

Timestamps

Timestamps denote the creation time on Zoom's server. The timestamp for each audio packet changes relative to the send_rate defined by the handshake request.

If the send_rate is set to 20ms (default), the timestamp for each audio packet will change by 20ms.

When working with streaming audio, timestamps are useful in determining the sequence of messages. Use timestamps to

• Infer the period of time where a user might be muted

User IDs and timestamps

When selecting multiple streams, your application will receive an audio stream for each participant in the session. By sending separate streams, RTMS enables your app to perform audio mixing, isolation, and individual analysis.

Each user in the session will have a unique user_id and their own incremental timestamp.

For merged audio, the user_id will be 0.

Buffered audio

When a session.rtms_started webhook event is received, the RTMS server starts buffering audio packets, and the timestamps start to increment, while the signaling connection is made. The RTMS server buffers audio up to 60 seconds while the signaling and media connections are established. Once the connections are established, the buffered audio packets are delivered.

To determine the amount of buffered data, calculate the difference between the timestamp of the rtms_started event and the first packet of audio data.

buffer_duration = firstPacketTimestamp - rtmsStartedEventTs

Best practices

When a session starts, capture the first timestamp from the signaling connection. This denotes the start of the session.

When participants mute their microphones, the RTMS server stops sending audio packets for that user. Use timestamps to detect these gaps and insert silence if needed for your application.

When working with pulse-coded modulation (PCM) audio consider the following:

• The size of raw PCM buffer and the storage it might take up without compression
• That you may need to convert the audio to WAV format to utilize services such as live streaming or speech to text transcription.
• That compression to lossy formats, such as mp3, requires the entire audio file to be completed before compression. We recommend you compress the audio after the session has ended.

Transcripts

Transcript data is available per participant with attribution, also called diarization.

Transcripts arrive continuously as speech is detected and processed in real time.

The RTMS server sends transcript data packets from each participant as text data with the participant's identifier (user_id), username (user_name), timestamp, and language.

Example transcript packet

{
    "msg_type": 17,
    "content": {
        "user_id": 19778240,
        "user_name": "John Smith",
        "timestamp": 1727384349000,
        "language": 9,
        "data": "Hi, hello world!"
    }
}

Languages

The language field provides the automatically detected language spoken in the session. When switching from one language to another, it typically takes 10-30 seconds for automatic detection to identify the new language.

Timestamps

Timestamps are sent when the sentence/utterance begins. Use this to create a log of when the user began speaking.

Best practices

As Zoom's transcription optimizes for low latency, it may be helpful to post-process text transcripts into final assets.

In sentence detection, pauses in speech can sometimes be mistaken as an end of a sentence.

Combining multiple transcript messages using message interval and timeouts is a useful strategy to combine disjointed sentences.