With Realtime Media Streams (RTMS) now available for Video SDK, you can access per-participant audio streams over WebSockets and process them on your server. While RTMS also allows you to directly access the participant's transcript data, in this blog we'll showcase how to access real-time audio and transcribe it with Zoom Scribe Live from your server. You can learn more about Realtime Media Streams in our RTMS docs.

Prerequisites#

• Node.js LTS
• A Zoom Video SDK account with universal credit enabled
• Zoom AI Services credentials with access to Scribe Live
• A service to tunnel your local server to a public URL (like Ngrok )

Enable RTMS for your Video SDK app#

Before building, you need to configure event subscriptions in the Zoom App Marketplace to receive RTMS lifecycle events.

1. Sign into the Zoom App Marketplace  with your Video SDK credentials
2. Navigate to Develop → Build Video SDK
3. Under Add feature, enable Event Subscriptions
4. Configure your subscription:
   • Add a descriptive name for your subscription
   • Add the RTMS Started and RTMS Stopped events
5. Set your Event notification endpoint URL - this is where Zoom will send webhook events when RTMS sessions start and stop
6. Save your configuration

You can use a service like ngrok to tunnel your local server to a public URL. Once installed you can run ngrok http 3000. Once configured, your server will receive webhook payloads when participants trigger RTMS in a Video SDK session:

Building the application#

You can find the completed project on GitHub . To follow along, create a new node project and initialize a package.json file:

npm init -y

Create a .env file in the root directory with your Zoom credentials:

ZM_RTMS_CLIENT=your_video_sdk_key
ZM_RTMS_SECRET=your_video_sdk_secret
ZOOM_SECRET_TOKEN=your_webhook_secret_token
ZOOM_API_KEY=your_ai_services_api_key
ZOOM_API_SECRET=your_ai_services_api_secret
VITE_PORT=3000

Install the dependencies:

npm install express dotenv ws

Scribe Live uses separate Zoom AI Services credentials. The sample signs a short-lived JWT with ZOOM_API_KEY and ZOOM_API_SECRET for each Scribe connection. You can also supply an AI Services bearer credential through ZOOM_ACCESS_TOKEN. Keep these credentials on the backend.

At a high level, the flow looks like this:

1. A webhook receives the RTMS start event from Zoom with connection details.
2. Our server establishes a signaling WebSocket connection and authenticates.
3. Upon successful handshake, we connect to the media WebSocket to receive audio data.
4. Audio packets are streamed to Scribe Live, which returns completed transcripts.

Sample app#

We'll walk through the key components of the sample app to understand how RTMS works with Video SDK. In the repo , the server code lives under server/ and is organized into a few files:

server/
├── index.ts       # Express server, webhook handler, RTMS signaling/media sockets
├── util.ts        # Video SDK session tokens and AI Services JWTs
└── scribe.ts      # Scribe Live connection, audio streaming, and transcript events

Building the application#

Let's walk through building the application step by step, following the structure of index.ts .

Setting up the Express server#

First, we create our Express server and import the dependencies in server/index.ts.

import http from 'http';
import crypto from 'crypto';
import dotenv from 'dotenv';
import WebSocket from 'ws';
import fs from 'fs';
import path from 'path';
import { ScribeSession } from './scribe.ts';
import express from 'express';
import { generateScribeToken, type SampleAudioPacket } from './util.ts';

dotenv.config({ quiet: true });
const PORT = process.env.VITE_PORT || 3000;
const ZoomSecretToken = process.env.ZOOM_SECRET_TOKEN as string;
const ZoomClientId = process.env.ZM_RTMS_CLIENT as string;
const ZoomClientSecret = process.env.ZM_RTMS_SECRET as string;

const app = express();
app.use(express.json());

We read Zoom credentials from environment variables and use them throughout the server for webhook validation and RTMS signatures.

To access the data stream from RTMS we can either use the RTMS SDK or handle the webhook & WebSocket connections manually.

Using the SDK#

You can use the rtms  SDK to connect to RTMS streams. The SDK provides a simple interface to connect to RTMS streams and receive audio and transcript data.

import rtms from '@zoom/rtms';

rtms.onWebhookEvent(async ({ payload }) => {
  const scribe = createScribeSession();
  await scribe.ready;
  const client = new rtms.Client();
  client.setAudioParams({ contentType: 2, codec: 1, sampleRate: 1, channel: 1, dataOpt: 1, duration: 100, frameSize: 1600 })
  client.onTranscriptData((data, size, timestamp, metadata) => console.log(`${metadata.userName}: ${data}`)); // transcript
  client.onAudioData((data, size, timestamp, metadata) => {
    scribe.sendAudio({
      ...metadata,
      user_id: metadata.userId,
      user_name: metadata.userName,
      timestamp,
      data: data.subarray(0, size).toString('base64'),
    });
  });
  client.join({ ...payload, client: ZoomClientId, secret: ZoomClientSecret });
});

You can directly access the transcript data from the onTranscriptData callback, but for transcription with Scribe we'll forward the onAudioData callback's audio, timestamp, and source metadata. The createScribeSession helper is defined below. If you're using the SDK you can skip to the Handling audio data for transcription section. Read on to learn how the webhooks and websockets work under the hood.

Handling webhook events#

The server needs to listen for webhook events. We'll handle three scenarios: webhook validation, RTMS session start, and RTMS session stop.

Webhook validation#

When you first configure your webhook URL in the Zoom Marketplace, Zoom validates it by sending a challenge:

app.post('/webhook', async (req, res) => {
  const { event, payload } = req.body;
  if (event === 'endpoint.url_validation' && payload?.plainToken) {
    const hash = crypto.createHmac('sha256', ZoomSecretToken)
      .update(payload.plainToken)
      .digest('hex');
    return res.json({
      plainToken: payload.plainToken,
      encryptedToken: hash,
    });
  }
  res.sendStatus(200);
  // ... handle other events below
});

We hash the plainToken using our secret token and return it as encryptedToken. This proves we own the webhook endpoint.

Starting an RTMS session#

When someone starts RTMS in a Video SDK session, Zoom sends a session.rtms_started event with connection details:

if (event === 'session.rtms_started') {
  const { session_id, rtms_stream_id, server_urls } = payload;
  console.log("Starting RTMS for session:", { payload });
  connectToSignalingWebSocket(session_id, rtms_stream_id, server_urls);
}
// ...

The payload contains:

• session_id - Unique identifier for the Video SDK session
• rtms_stream_id - Unique identifier for this RTMS stream
• server_urls - Signaling WebSocket URL to connect to

We call connectToSignalingWebSocket with these values to establish the first WebSocket connection.

WebSocket connections#

RTMS uses two WebSocket connections: one for signaling and one for media data. Here are the events we will use for our app:

Both connections require an HMAC-SHA256 signature for authentication using the format CLIENT_ID,session_id,rtms_stream_id signed with your client secret. This is implemented in server/index.ts:

function generateSignature(sessionID: string, streamId: string): string {
    const message = `${ZoomClientId},${sessionID},${streamId}`;
    return crypto.createHmac('sha256', ZoomClientSecret).update(message).digest('hex');
}

Signaling connection#

The connectToSignalingWebSocket function (in server/index.ts) establishes the signaling connection and sends the initial handshake. Here's a simplified version:

function connectToSignalingWebSocket(session_id: string, rtmsStreamId: string, serverUrls: string ) {
  const signalingWs = new WebSocket(serverUrls, [], {rejectUnauthorized: false});
  signalingWs.on("open", () => {
      signalingWs.send(
        JSON.stringify({
        msg_type: 1,
        meeting_uuid: session_id,
        session_id,
        rtms_stream_id: rtmsStreamId,
        signature: generateSignature(session_id, rtmsStreamId),
      }))
  });

  signalingWs.on("message", (data) => {
    const msg = JSON.parse(data.toString());
    if (msg.msg_type === 12) { // KEEP_ALIVE_REQ
      signalingWs.send(
        JSON.stringify({
          msg_type: 13, // KEEP_ALIVE_RESP
          timestamp: msg.timestamp,
        }),
      );
    } else if (msg.msg_type === 2) { // SIGNALING_HAND_SHAKE_RESP
      if (msg.status_code === 0) {
        const mediaUrl = msg.media_server?.server_urls?.audio;
        connectToMediaWebSocket(
          mediaUrl,
          session_id,
          rtmsStreamId,
          signalingWs,
        );
      }
    }
  });
}

When the signaling handshake succeeds (receiving msg_type: 2 with status_code: 0), the signaling connection extracts the media server URL and calls connectToMediaWebSocket.

Media connection#

After the signaling handshake, the app opens a second WebSocket to the media server. It sends a handshake message specifying which types of media data it wants to receive. The flow is very similar to the first:

async function connectToMediaWebSocket(mediaUrl: string, session_id: string, rtmsStreamId: string, signalingSocket: WebSocket) {
  const scribe = createScribeSession();
  await scribe.ready;
  const mediaWs = new WebSocket(mediaUrl, [], { rejectUnauthorized: false });

  mediaWs.on("open", () => {
    const handshakeMsg = {
      msg_type: 3, // DATA_HAND_SHAKE_REQ
      protocol_version: 1,
      sequence: 0,
      meeting_uuid: session_id,
      rtms_stream_id: rtmsStreamId,
      signature: generateSignature(session_id, rtmsStreamId),
      media_type: 1, // AUDIO
      payload_encryption: false,
      media_params: {
        audio: {
          content_type: 2, // RAW_AUDIO for PCM/L16
          sample_rate: 1, //16k
          channel: 1, //mono
          codec: 1, //L16
          data_opt: 1, //AUDIO_MIXED_STREAM
          send_rate: 100, //in Milliseconds
        },
      },
    };

    mediaWs.on("message", (data) => {
      const msg = JSON.parse(data.toString());
      if (msg.msg_type === 14) { // AUDIO DATA
        if (msg.content?.data) {
          scribe.sendAudio(msg.content as SampleAudioPacket['content']);
        }
      } else if (msg.msg_type === 4 && msg.status_code === 0) { // DATA_HAND_SHAKE_RESP
        signalingSocket.send(
          JSON.stringify({
            msg_type: 7, // CLIENT_READY_ACK
            rtms_stream_id: rtmsStreamId,
          }),
        );
      } else if (msg.msg_type === 12) { // KEEP_ALIVE_REQ
        mediaWs.send(
          JSON.stringify({
            msg_type: 13, // KEEP_ALIVE_ACK
            timestamp: msg.timestamp,
          }),
        );
      }
    });
    mediaWs.send(JSON.stringify(handshakeMsg));
  });
}

We get the audio data on the message event of the media WebSocket. Audio data is sent as uncompressed raw PCM (L16) data with a 16kHz sample rate and a mono channel. The media_type: 1 value indicates we want only audio. PCM/L16 requires raw audio (content_type: 2), not RTP. Learn more: media_data_type.

Processing media messages#

Incoming packets on the media WebSocket are handled inline in server/index.ts. The message handler responds to:

• 4 (DATA_HAND_SHAKE_RESP): Media handshake response. If successful (status_code === 0), the server sends a CLIENT_READY_ACK (msg_type: 7) to the signaling socket.
• 12 (KEEP_ALIVE_REQ): Media server keep-alive ping. The client must reply with KEEP_ALIVE_RESP (msg_type: 13) and the provided timestamp.
• 14 (AUDIO): Audio packets encoded as base64 PCM data. The code passes the packet to scribe.sendAudio, which decodes the audio and forwards it with its timestamp and metadata.

The handler sends the required acknowledgements and forwards audio to Scribe. We'll walk through the transcription connection in the next section.

Stopping an RTMS session#

When the RTMS session ends, we clean up our connections:

else if (event === 'session.rtms_stopped') {
  const { session_id } = payload;
  console.log(`Stopping RTMS for Video session ${session_id}`);
}

The complete sample closes the matching RTMS connections and calls scribe.close() on stop or interruption. This sends session.close and keeps reading Scribe results until session.closed, with a 10-second shutdown timeout.

Handling audio data for transcription#

When we receive audio packets (msg_type: 14), we stream their raw PCM data to Scribe Live instead of converting them to WAV files. The connection and audio handling live in server/scribe.ts.

Connecting to Scribe Live#

The createScribeSession helper creates the sample's Scribe wrapper and saves completed transcripts:

function createScribeSession() {
  return new ScribeSession({
    token: process.env.ZOOM_ACCESS_TOKEN || generateScribeToken(
      process.env.ZOOM_API_KEY!,
      process.env.ZOOM_API_SECRET!,
    ),
    onTranscript: (text) => {
      console.log(text);
      fs.appendFileSync(path.join(process.cwd(), 'server', 'transcript.txt'), `${text}\n`);
    },
    onFailure: () => console.error('Scribe connection failed; close the matching RTMS stream.'),
  });
}

This helper illustrates session creation; the complete sample also closes the affected RTMS connection on failure. Each RTMS stream gets its own Scribe session. The wrapper connects to wss://api.zoom.us/v2/aiservices/scribe/live with the live-asr subprotocol and an Authorization: Bearer header. When the socket opens, it sends:

socket.send(JSON.stringify({
  type: 'session.update',
  config: { language: 'en-US', model: 'zoom-scribe-multilingual' },
  audio: { format: 'pcm16' },
}));

The wrapper resolves scribe.ready after receiving session.updated. Waiting for this before connecting the RTMS media stream ensures Scribe is configured when audio arrives. See server/scribe.ts  for the full implementation.

Streaming and transcribing#

RTMS sends Base64-encoded raw PCM (L16) at 16 kHz, mono. Scribe receives binary PCM16 frames without a WAV header. The wrapper decodes each packet and splits it into frames of at most 3,200 bytes, or 100 ms of audio:

const { data, timestamp, ...metadata } = content;
const audio = Buffer.from(data, 'base64');
for (let offset = 0; offset < audio.length; offset += 3200) {
  socket.send(JSON.stringify({
    type: 'input_audio_buffer.append',
    timestamp,
    metadata,
  }));
  socket.send(audio.subarray(offset, offset + 3200));
}

Each metadata message immediately precedes its audio frame. Source identifiers and custom fields pass through as metadata, including optional user_id, user_name, or channel_id. Mixed audio keeps its source identity, including user_id: 0; do not substitute the active speaker's identity.

The wrapper handles completed transcripts returned by Scribe:

socket.on('message', (data) => {
  const event = JSON.parse(data.toString());
  if (event.type === 'transcription.completed') {
    const text = event.transcript?.trim();
    if (text) options.onTranscript(text);
  }
});

The callback logs each completed result and appends it to server/transcript.txt. The sample stores transcript text only; Scribe can also return the associated source metadata. When RTMS omits audio during pauses, the wrapper sends silent PCM frames to keep the connection active. Learn more in the Scribe Live documentation.

Running the server#

Finally we start the HTTP server in index.ts file:

const server = http.createServer(app);

server.listen(PORT, () => {
  console.log(`Server running at http://localhost:${PORT}`);
});

Start the RTMS session from your server#

Now that the server is configured, you can use the REST API to start the RTMS streams:

fetch(`https://api.zoom.us/v2/videosdk/sessions/${sessionId}/rtms_app/status`, {
  method: 'PATCH',
  headers: {
    'Content-Type': 'application/json',
    Authorization: 'Bearer YOUR_VIDEO_SDK_API_JWT'
  },
  body: JSON.stringify({
    action: 'start'
  })
})

Alternatively, you can also use the Video SDK RealTimeMediaStreamsClient  object to start/stop the RTMS streams.

Conclusion#

With RTMS and Video SDK, you can build server-side applications that process real-time media without running automated clients. With Scribe Live handling transcription, you could use the resulting text for real-time translation, speech analytics, or meeting summarization.

Check out the RTMS documentation and explore more samples in our rtms-samples repository .