Use raw data

Access raw video and raw audio data through raw recording or raw streaming. When accessing raw data through raw recording, the host gives the meeting participant associated with the SDK app local recording permission. When accessing raw data through raw streaming, the host gives the meeting participant associated with the SDK app live streaming permission.

This enables more flexibility than either local recording or live streaming for video and audio streams. In addition to session-wide audio and video access, this enables access to raw audio and video data for each participant unless the participant's audio or video are muted.

For raw recording, if the SDK user is also the meeting host, they can start both raw recording and an actual local recording. If the SDK user is not the meeting host, they must request local recording permission from the meeting host. In a meeting, the host can perform cloud recording, with one or more SDK clients doing raw recording locally.

For raw streaming, if the SDK user is also the meeting host, they can start both raw streaming and an actual live streaming service. If the SDK user is not the meeting host, they must request raw streaming permission from the meeting host. In a meeting, the host can perform real external live streaming, with one or more SDK clients doing raw streaming locally.

The SDKs prompt users with a dialog to let them know when recording or streaming has started, as in the Zoom clients used by other meeting participants.

Raw data formats#

• Video is in YUV420p format
• Audio is in PCM format

Before you start#

To work with raw data in Meeting SDK for Windows, you must:

• Have Meeting SDK for Windows version 5.9.0 or later
  • Raw streaming by meeting host: Meeting SDK for Windows version 5.11.0 or later
  • Raw streaming by non-host participant: Meeting SDK for Windows version 5.12.8 or later
  • To request local recording permission from host: Meeting SDK for Windows version 5.13.5 or later

• Meet these requirements for using raw recording
  • The meeting must enable local recording 
  • Meet any one of these additional requirements
    • You are a normal participant but with local recording permission 
    • You are the meeting host or co-host

• Meet all of these requirements for raw streaming
  • The current user has raw live streaming permission
  • The meeting host has a Pro, Business, Education, or Enterprise account
  • The meeting host is licensed 

Recording and streaming sequence diagrams#

This sequence diagram shows the raw recording flow.

This sequence diagram shows the raw streaming flow.

Usage#

• Set up video raw data callbacks
• Set up audio raw data callbacks
• Request LocalRecording permission
• Call StartRawRecording to start recording raw data
• Call StopRawRecording to stop recording raw data
  OR
• Request RawLiveStream permission
• Call StartRawLiveStream to start streaming raw data
• Call StopRawLiveStream to stop streaming raw data

Methods#

• Video raw data callbacks : onRawDataFrameReceived — triggered for each video frame
• Audio raw data callbacks 
  • onOneWayAudioRawDataReceived - per-user audio (no support for telephone participants)
  • onMixedAudioRawDataReceived - session-wide audio (what meeting participants hear)
• StartRawRecording  — start raw recording
• StopRawRecording  — stop raw recording
• StartRawLiveStream  — start raw streaming
• StopRawLiveStream  — stop raw streaming

Record raw data#

Follow these steps to record audio, video, or a screen.

1. Implement an instance of IZoomSDKRendererDelegate.

   • In this class, respond to onRawDataFrameReceived to process the raw video or raw screen data stored in the YUVRawDataI420 object.
   • Similarly, set up an IZoomSDKAudioRawDataDelegate for the audio raw data.
   • There are two different callbacks that can be used to receive raw data: onOneWayAudioRawDataReceived for per-user audio and onMixedAudioRawDataReceived for meeting-wide audio.

   Note: Do not perform heavy operations from within the raw data callbacks. This may cause delayed or lost data.

2. If the current user has permission to start the recording, call StartRawRecording in IMeetingRecordingController to trigger the recording consent dialog for all users.

   • A user can either request local recording permission by asking the meeting host or by joining the meeting with a special local recording token (see instructions below.)
   • A raw recording is a type of local recording, so listen for the onRecordingStatus callback to determine when the recording successfully starts.

3. Once the recording starts, implement an instance of IZoomSDKRenderer. Pass the IZoomRendererDelegate and IZoomSDKRenderer into the createRenderer method in zoom_rawdata_api.h.

   • For audio, pass your delegate into the IZoomSDKAudioRawDataHelper class subscribe function.
   • For raw video, call the subscribe(UserID, RAW_DATA_TYPE_VIDEO) method in IZoomSDKRenderer.
   • For raw share, call the subscribe(UserID, RAW_DATA_TYPE_SHARE) method in IZoomSDKRenderer.

4. To end the recording, call StopRawRecording in IMeetingRecordingController.

   • Unsubscribe in any IZoomSDKRenderer and IZoomSDKRawDataHelper instance(s) you used.

Receive local recording permission through an OAuth app#

To successfully request recording permission, your OAuth app must be:

• a public OAuth app in the Zoom Marketplace
• installed by the meeting host or by their account admin

1. Your OAuth app requests one of the two scopes:
   • Meeting_token:read:admin:local_recording - account level, installed by admin
   • Meeting_token:read:local_recording - user level, installed by host
2. Once the SDK app has the meeting ID it needs to join, have it call the "Get a meeting's join token for local recording" REST API endpoint
3. Have your OAuth app pass the returned token to your Meeting SDK app.
4. Have the Meeting SDK app use the returned token to join the meeting with the app_privilege_token  parameter.

Stream raw data from within the current meeting#

Follow these steps to stream either audio or video from inside a meeting.

1. Implement an instance of IZoomSDKRendererDelegate.

   • In this class, respond to onRawDataFrameReceived to process the raw video data stored in the YUVRawDataI420 object.
   • Additionally, set up an IZoomSDKAudioRawDataDelegate for the raw audio data.
   • Use one of two different callbacks to receive raw data onOneWayAudioRawDataReceived for per-user audio, or onMixedAudioRawDataReceived for meeting-wide audio.

   Note: Do not perform heavy operations from within the raw data callbacks. This may cause delayed or lost data.

2. Call StartRawLiveStream in IMeetingLiveStreamController to trigger the live streaming consent dialog for all users, if the current user has permission to start the live streaming.

   • A raw stream is a type of live streaming, so listen for the onUserRawLiveStreamingStatusChanged  callback to determine when the streaming successfully starts.

3. Once the streaming starts, pass the IZoomRendererDelegate into the createRenderer method in zoom_rawdata_api.h.

   • For audio, pass your delegate to the IZoomSDKAudioRawDataHelper class's subscribe function.

4. To end the recording, call StopRawLiveStream in IMeetingLiveStreamController.

   • Unsubscribe in any IZoomSDKRenderer or IZoomSDKRawDataHelper instance you used.

Notify users when beginning a live stream#

When beginning a live stream inside a current meeting, display a notice telling users that the meeting will be streamed to another "platform" - the application or services that need the raw data - and identify the platform. This gives users an informed chance to opt out of being streamed on that new platform.

• RawLiveStreamInfo.broadcastName - the name of the app opening the new streaming session, such as "AI Note-taker by XYZ."
• RawLiveStreamInfo.broadcastUrl - the URL of a landing page hosted by the app developer OR the URL of the app's Zoom Marketplace page, such as https://example.com/description-page or https://marketplace.zoom.us/apps/XXXXXXexampleXXXX.

Request live streaming permission#

If you are not the meeting host, and do not have live streaming permission, you can either request this permission from the meeting host during the meeting or request live streaming permission through the OAuth app, so that the SDK client can start and stop raw streaming.

Request streaming permission from the host#

To request live streaming permission from the host during the meeting:

1. Call RequestRawLiveStream().
2. Receive permission by checking the callback onRawLiveStreamPrivilegeChanged().

Request streaming permission through an OAuth app#

To successfully request live-streaming permission through an OAuth app, your app must be:

• a public OAuth app on the Zoom Marketplace
• installed by the meeting host or by their account admin

1. Your OAuth app will request one of the two scopes:
   • Meeting_token:read:admin:live_streaming - account level, installed by admin
   • Meeting_token:read:live_streaming - user level, installed by host
2. Once your SDK app has the meeting ID it needs to join, it calls the "Get a meeting's join token for live streaming" REST API endpoint
3. Have the OAuth app pass the returned token to your Meeting SDK app.
4. Have the Meeting SDK app use the returned token to join the meeting with the app_privilege_token  parameter.

Share raw data#

Follow these steps to share audio, video, or a screen.

Share audio in PCM buffer format#

1. Implement an instance of IZoomSDKVirtualAudioMicEvent.
   • Save a reference to the method parameter IZoomSDKAudioRawDataSender pSender in the onMicInitialize method.
   • Respond to onMicStartSend. Using your reference to pSender, call send(char* data, unsigned int data_length, int sample_rate).
2. Call GetAudioRawdataHelper, which returns the IZoomSDKAudioRawDataHelper object.
3. Pass an implementation of IZoomSDKVirtualAudioMicEvent into the setExternalAudioSource(MyIZoomSDKVirtualAudioMicEvent) method, found in the IZoomSDKAudioRawDataHelper object.

This invokes onMicStartSend. You can now send audio in PCM buffer format.

Share video in YUV420 buffer format#

1. Implement an instance of IZoomSDKVideoSource.
   • Save a reference to the method parameter IZoomSDKVideoSender sender found in onInitialize method.
   • Respond to onStartSend. Using your reference to sender, call sendVideoFrame(char* frameBuffer, int width, int height, int frameLength, int rotation).
2. Call GetRawdataVideoSourceHelper, which returns the IZoomSDKVideoSourceHelper object.
3. Pass the implementation of IZoomSDKVideoSource into the setExternalVideoSource(MyIZoomSDKVideoSource) method, found in the IZoomSDKVideoSourceHelper object.

This invokes onStartSend. You can now send video in YUV420 buffer format.

Share a screen in YUV420#

1. Implement an instance of IZoomSDKShareSource.
   • Respond to onStartSend to send video in YUV420 buffer format. Use (IZoomSDKShareSender* pSender) in the method parameter to call sendFrame(char* frameBuffer, int width, int height, int frameLength, int rotation).
2. Call GetRawdataShareSourceHelper(), which returns the IZoomSDKShareSourceHelper object.
3. Pass the implementation of IZoomSDKShareSource into the setExternalShareSource(MyIZoomSDKShareSource) method, found in the IZoomSDKShareSourceHelper object.

This invokes onStartSend. You can now screen-share in YUV420 buffer format.

Notes#

• The usage or actual handling of the raw data may vary based on your use case.
• The SDK only provides the raw data. Implement only the raw data video or audio callbacks that you need.
• Performing any heavy handling within the callback delays the callback being triggered, which may result in loss of data.