# Use OpenAI Voice Agents with the Zoom Video SDK

<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

[OpenAI Voice Agents](https://platform.openai.com/docs/guides/voice-agents) can listen to user audio, understand intent, and respond conversationally in real time. They're a great way to add AI to your apps while maintaining a natural, conversational interaction for your users.

In this guide, we’ll implement a speech-to-speech (S2S) architecture — where audio is processed and generated directly, without depending on text transcripts. This design enables low-latency, natural conversations that feel fluid and responsive.

We’ll use the [OpenAI Agents SDK](https://platform.openai.com/docs/guides/agents-sdk) & [Zoom Video SDK](/docs/video-sdk/) to make the integration simple. If you’re new to the Zoom SDK, start with our [Video SDK Next.js guide](/blog/nextjs-video-conferencing-app-using-the-zoom-video-sdk/). For this demo, we’ll build upon the official OpenAI [realtime-agents sample app](https://github.com/openai/openai-realtime-agents/). We'll showcase the steps needed to integrate the Zoom Video SDK. You can find the complete app on [GitHub](https://github.com/zoom/videosdk-openai-realtime-voiceagents).

## Prerequisites

-   Node & NPM
-   Zoom Video SDK [account](/docs/video-sdk/developer-accounts/) and [credentials](/docs/video-sdk/get-credentials)
-   OpenAI account & `OPENAI_API_KEY` to use Realtime Agents

1. Clone the OpenAI sample:

```bash
git clone https://github.com/openai/openai-realtime-agents.git
```

2. Instal the dependencies for the project:

```bash
npm install @zoom/videosdk jsrsasign @types/jsrsasign lucide-react
```

We'll install `@zoom/videosdk` to use the Zoom Video SDK. We'll also install `jsrsasign` to create & sign JWTs and `lucide-react` to use the Lucide icons.

3. Add environment variables:

```bash
cp .env.sample .env
```

Set the ZOOM_SDK_KEY, ZOOM_SDK_SECRET, OPENAI_API_KEY as:

```
OPENAI_API_KEY=sk-proj-xxxx
ZOOM_SDK_KEY=sdkkey123
ZOOM_SDK_SECRET=secret123
```

## What we'll build

-   Generate a JSON Web Token (JWT) on the server for the Zoom Video SDK
-   Join a Video SDK session and render user videos
-   Automatically connect or disconnect the OpenAI Realtime agent when a Video SDK session is joined/left
-   Automatically mute the user audio when using push-to-talk to talk to the voice agent

## Generate a Zoom SDK JWT (server)

The Zoom SDK uses JSON Web Tokens (JWTs) to authenticate users. We'll create a server-only utility to create a JWT using your Zoom SDK Key & Secret. You can find these credentials in your account in the [Zoom Marketplace](https://marketplace.zoom.us/).

Create a new file `src/data/getToken.ts`. We'll export an async function `getData` that will return the JWT to our component:

```ts
// src/data/getToken.ts
import "server-only";
import { KJUR } from "jsrsasign";

export async function getData(slug: string) {
    const JWT = await generateSignature(slug, 1);
    return JWT;
}
```

To generate a JWT, we'll define a `generateSignature` function and use the `jsrsasign` library to sign the JWT:

```ts
..

function generateSignature(sessionName: string, role: number) {
  if (!process.env.ZOOM_SDK_KEY || !process.env.ZOOM_SDK_SECRET) {
    throw new Error("Missing ZOOM_SDK_KEY or ZOOM_SDK_SECRET");
  }
  const iat = Math.round(new Date().getTime() / 1000) - 30;
  const exp = iat + 60 * 60 * 2;
  const oHeader = { alg: "HS256", typ: "JWT" };
  const sdkKey = process.env.ZOOM_SDK_KEY;
  const sdkSecret = process.env.ZOOM_SDK_SECRET;
  const oPayload = {
    app_key: sdkKey, tpc: sessionName, role_type: role, version: 1, iat: iat, exp: exp,
  };

  const sHeader = JSON.stringify(oHeader);
  const sPayload = JSON.stringify(oPayload);
  const sdkJWT = KJUR.jws.JWS.sign("HS256", sHeader, sPayload, sdkSecret);
  return sdkJWT;
}
```

We can fetch this JWT in the Next.js base route in `src/app/page.tsx`:

```tsx
// src/app/page.tsx
import React, { Suspense } from "react";
import { getData } from "@/data/getToken";
import App from "./App";

export default async function Page() {
  const jwt = await getData("test");
  return (
    <Suspense fallback={Loading...}>
      <App jwt={jwt} />
    </Suspense>
  );
}
```

## Join a Zoom Video SDK session and render video

We'll create a `Videochat` component that initializes the Zoom client, joins a Video SDK session using the JWT, and renders videos when peers start/stop.

Create a new file `src/app/zoom/Videochat.tsx`:

```tsx
// src/app/zoom/Videochat.tsx
import ZoomVideo, { type VideoClient, VideoQuality, type VideoPlayer } from "@zoom/videosdk";

const Videochat = (props: { slug: string; JWT: string }) => {
  const session = props.slug;
  const jwt = props.JWT;
  const [inSession, setInSession] = useState(false);
  const client = useRef<typeof VideoClient>(ZoomVideo.createClient());
  const [isVideoMuted, setIsVideoMuted] = useState(!client.current.getCurrentUserInfo()?.bVideoOn);
  const [isAudioMuted, setIsAudioMuted] = useState(client.current.getCurrentUserInfo()?.muted ?? true);
  const videoContainerRef = useRef<HTMLDivElement>(null);
```

The code here is very similiar to the Next.js sample app. We'll initialize the Zoom client and our app state:

```ts
const joinSession = async () => {
    await client.current.init("en-US", "Global", { patchJsMedia: true });
    client.current.on("peer-video-state-change", renderVideo);
    await client.current.join(session, jwt, userName);
    setInSession(true);
    const mediaStream = client.current.getMediaStream();
    await mediaStream.startAudio();
    setIsAudioMuted(mediaStream.isAudioMuted());
    await mediaStream.startVideo();
    setIsVideoMuted(!mediaStream.isCapturingVideo());
    await renderVideo({
        action: "Start",
        userId: client.current.getCurrentUserInfo().userId,
    });
};
```

We can access the media stream from the client instance and start the audio and video by calling the `startAudio` and `startVideo` methods. We'll update the local user state by setting the `isAudioMuted` and `isVideoMuted` states to the current user's audio and video state. We'll call the `renderVideo` function to update the videos on screen. We'll define the `renderVideo` function next.

The `renderVideo` function will attach/detach the video stream for the user when they start/stop video:

```ts
const renderVideo = async (event: {
    action: "Start" | "Stop";
    userId: number;
}) => {
    const mediaStream = client.current.getMediaStream();
    if (event.action === "Stop") {
        const element = await mediaStream.detachVideo(event.userId);
        if (Array.isArray(element)) element.forEach((el) => el.remove());
        else element.remove();
    } else {
        const userVideo = await mediaStream.attachVideo(
            event.userId,
            VideoQuality.Video_360P,
        );
        videoContainerRef.current!.appendChild(userVideo as VideoPlayer);
    }
};
```

Finally the `leaveSession` will leave the session and clear the state:

```ts
  const leaveSession = async () => {
    client.current.off("peer-video-state-change", renderVideo);
    await client.current.leave();
    window.location.href = "/";
  };
};
```

The Zoom SDK needs to access browser APIs (like the window object), these aren't available server-side in Next.js. So, we load the `Videochat` component client-only with a tiny wrapper in `src/app/zoom/VideochatClientWrapper.tsx`:

```tsx
// src/app/zoom/VideochatClientWrapper.tsx
import dynamic from "next/dynamic";
const Videochat = dynamic<{ slug: string; JWT: string }>(
    () => import("./Videochat"),
    { ssr: false },
);
export default function VideochatClientWrapper({
    slug,
    JWT,
}: {
    slug: string;
    JWT: string;
}) {
    return <Videochat slug={slug} JWT={JWT} />;
}
```

## Sync the OpenAI session with Zoom session

Inside `App.tsx`, the demo already has `connectToRealtime` & `disconnectFromRealtime` functions. We can re-use these function to sync the OpenAI session with the Video SDK session.
We can subscribe to Zoom `connection-change` event and call these functions. We'll create a useEffect to attach an event listener to trigger them on `connection-change`:

```tsx
useEffect(() => {
    if (typeof window !== "undefined") {
        import("@zoom/videosdk").then(
            ({ default: ZoomVideo, ConnectionState }) => {
                const client = ZoomVideo.createClient();
                clientRef.current = client;
                client.on("connection-change", (e) => {
                    if (e.state === ConnectionState.Connected) {
                        if (sessionStatus === "DISCONNECTED")
                            connectToRealtime();
                    }
                    if (e.state === ConnectionState.Closed) {
                        disconnectFromRealtime();
                    }
                });
            },
        );
    }
}, []);
```

## Push-to-Talk (PTT): mute mic on Zoom while sending audio to the agent

In sessions where multiple users are communicating, it can be nice to have a voice agent that a user can ask questions, or use to takes notes. We can implement PTT to mute the user's mic in the Video SDK session while they're talking to the agent. Releasing the "Talk" button will unmute the Zoom mic.

To do this, we already have the `handleTalkButtonUp` and `handleTalkButtonDown` functions in the `App` component, we'll modify them to add the Video SDK mute/unmute logic:

```diff
const handleTalkButtonDown = () => {
  if (sessionStatus !== 'CONNECTED') return;
+ if (clientRef.current) {
+   const stream = clientRef.current.getMediaStream();
+   stream.muteAudio()
+ }
  interrupt();
  setIsPTTUserSpeaking(true);
  sendClientEvent({ type: 'input_audio_buffer.clear' }, 'clear PTT buffer');
};

const handleTalkButtonUp = () => {
  if (sessionStatus !== 'CONNECTED' || !isPTTUserSpeaking)
    return;
+ if (clientRef.current) {
+   const stream = clientRef.current.getMediaStream();
+   stream.unmuteAudio()
+ }
  setIsPTTUserSpeaking(false);
  sendClientEvent({ type: 'input_audio_buffer.commit' }, 'commit PTT');
  sendClientEvent({ type: 'response.create' }, 'trigger response PTT');
};
```

That's all the code we need to add the Zoom Video SDK. Let's take a look at the high level architecture of the OpenAI sample to understand how it works:

## The OpenAI Agents SDK architecture

-   **Ephemeral key (server → client)**: The browser calls `GET /api/session` to fetch a short‑lived token. This keeps your OpenAI credentials off the client while letting the browser connect directly to the Realtime API.
-   **Create a realtime session**: The app builds a `RealtimeSession` with a WebRTC transport, picks a realtime model, and sets audio input/output formats.
-   **Pick an agent**: Agents live in `src/app/agentConfigs/`. You pass one as `initialAgent`—it defines the agent’s voice, instructions, and optional tools it can call during the conversation.
-   **Connect and stream**: `session.connect()` completes the WebRTC handshake and opens a data channel for low‑latency events. From here, audio flows both ways in real time.
-   **Sending data and events**: The `session` object (stored inside a React ref) gives you methods to `sendMessage`, `sendAudio`, `interrupt`, `mute`, `pushToTalkStart`, `pushToTalkStop`, and more.
-   **Send input, get responses**: User audio is send to the model and the response is played back to the user.

## Running the app

To run the app, you can use the following command:

```bash
bun dev
```

This will start the app on `http://localhost:3000`.

With these additions, you now have an integrated Zoom Video SDK experience with OpenAI Realtime Agents, complete with synchronized session lifecycles and push-to-talk mic control. You can now start a session and talk to the voice agent.

## Conclusion

In this guide, we've added the Zoom Video SDK to the OpenAI sample app. This opens up many new voice agent use-cases in voice and video chat apps. To learn more about the Video SDK, check out our [documentation](/docs/video-sdk/web/). We're excited to see what you'll build!