Raw data

The Video SDK provides you with an option to access real-time raw audio, video, and share data during a session, so you can apply additional effects and enhance the user experience. Use a processor component to apply effects or transformations to each piece of data in real-time.

An example of raw share data is when a user shares their screen.

  • Receive and send raw share video data using the same logic as video data, except get the share pipe instead of the video pipe in the raw data pipe method.
  • Receive and send raw share audio data using the same method used for raw audio data.

Follow the process outlined in this flowchart to capture and process raw data.

Flowchart outlining the raw data flow

Select raw data memory mode

In order to utilize raw data of any type, you must first select the memory mode you wish to use. The SDK supports heap-based and stack-based memory modes.

Stack-based memory

  • Variables are allocated automatically and deallocated after the data leaves scope.
  • Variables are not accessible from or transferable to other threads.
  • Typically has faster access than heap-based memory allocation.
  • Memory space is managed by the CPU and will not become fragmented.
  • Variables cannot be resized.

See Stack-based memory allocation for details.

Specify with:

ZoomVideoSDKRawDataMemoryModeStack

Heap-based memory

  • Variables are allocated and deallocated manually. You must allocate and free variables to avoid memory leaks.
  • Variables can be accessed globally.
  • Has relatively slower access than stack-based memory allocation.
  • Has no guarantee on the efficiency of memory space and can become fragmented.
  • Variables can be resized.

See Heap-based dynamic memory allocation for details.

Specify with:

ZoomVideoSDKRawDataMemoryModeHeap

Specify memory mode

After determining which memory mode is right for you, you must specify it when the SDK is initialized. Note that this must be done for audio. video, and share individually. To specify a raw data memory mode, provide one of these enums cases to the ZoomVideoSDKInitParams during SDK initialization.

ZoomVideoSDKInitParams initParams;
// Set audio memory mode to stack.
initParams.audioRawDataMemoryMode = ZoomVideoSDKRawDataMemoryModeStack;
// Set video memory mode to stack.
initParams.videoRawDataMemoryMode = ZoomVideoSDKRawDataMemoryModeStack;
// Set share memory mode to stack.
initParams.shareRawDataMemoryMode = ZoomVideoSDKRawDataMemoryModeStack;

To use heap-based memory, use ZoomVideoSDKRawDataMemoryModeHeap.

Receive raw video data

Raw video and share data is encoded in the YUV420p format. YUV420 is a data object commonly used by the renderer based on OpenGL ES. To access and modify the video data, you will need to do the following:

  1. Implement an instance of IZoomVideoSDKRawDataPipeDelegate.
  2. Use callback functions provided by IZoomVideoSDKRawDataPipeDelegate to receive each frame of the raw video data.
  3. Pass the delegate into the video pipe of a specific user.
ZoomVideoSDKRawDataPipeDelegate *dataDelegate = new ZoomVideoSDKRawDataPipeDelegate();
void ZoomVideoSDKRawDataPipeDelegate::onRawDataFrameReceived(YUVRawDataI420 *data) {
    // Add your code, for example, to store the raw data to a file, send it to be processed, etc.
}
void ZoomVideoSDKRawDataPipeDelegate::onRawDataStatusChanged(RawDataStatus status) {
    // Add your code to process this.
}
ZoomVideoSDKRawDataPipe pipe = user.getVideoPipe();
pipe.subscribe(ZoomVideoSDKVideoResolution.VideoResolution_360P, dataDelegate);

Each frame of video data will be made available through the YUVRawDataI420 object. Various pieces of data can be accessed through this object in onRawDataFrameReceived:

const int width = data->GetStreamWidth();
const int height = data->GetStreamHeight();
const int bufLen = data->GetBufferLen();
const int rotation = data->GetRotation();
const int sourceID = data->GetSourceID();

The Video SDK for Windows supports receiving videos in the resolutions enumerated in ZoomSDKVideoResolution. You can find more information in the SDK reference.

Receive raw audio data

Through your implementation of IZoomVideoSDKDelegate, you can access mixed (combined audio output from one or more users, as heard in a session) and per-user raw audio data.

Unlike raw video data, raw audio data will default to stack-based memory if you do not specify a memory mode.

  • The virtual speaker allows access to audio data received from other users in the session. This data represents what a user would hear played through the device's speakers.
  • The virtual mic allows audio data for the current user to be sent to the session programmatically instead of from the SDK capturing it through an audio input device.

You can receive raw audio if it was sent through IZoomVideoSDKVirtualAudioMic. To do so:

  1. Implement an instance of IZoomVideoSDKVirtualAudioSpeaker.
  2. Pass that implementation into ZoomVideoSDKSessionContext.
  3. Access raw data in each callback method.
ZoomVideoSDKVirtualAudioSpeaker *vSpeaker = new ZoomVideoSDKVirtualAudioSpeaker();
session_context.virtualAudioSpeaker = vSpeaker;

To access raw audio data, you must:

  1. Access IZoomVideoSDKAudioHelper using getAudioHelper();.
  2. startAudio using IZoomVideoSDKAudioHelper.
  3. Subscribe to audio using IZoomVideoSDKAudioHelper.
  4. Listen for the following callbacks in your listener:
IZoomVideoSDKAudioHelper *m_pAudiohelper = video_sdk_obj->getAudioHelper();
m_pAudiohelper->startAudio();
m_pAudiohelper->subscribe();
virtual void onMixedAudioRawDataReceived(AudioRawData *data_){
    // Access mixed data for the whole session here
};
virtual void onOneWayAudioRawDataReceived(AudioRawData *data_, IZoomVideoSDKUser *pUser){
    // Access user-specific raw data here for the user associated with the userId
};

Receive raw share data

Receive raw share video and audio data, for example, when someone is sharing their screen, similarly to how you receive raw video and raw audio data.

Receive raw share video data

Follow the same steps and code to receive raw video data, except get the share data instead of the video data, see the following code for an example.

// Previous code same as receive raw video
ZoomVideoSDKRawDataPipe pipe = user.GetSharePipe();
pipe.subscribe(ZoomVideoSDKVideoResolution.VideoResolution_360P, dataDelegate);

Receive raw share audio data

Receive raw share audio data in the same way as you receive raw audio data.

Send raw video data

In addition to receiving and processing raw video data, you can also send raw or processed video data of a user from the user's device. To do so:

  1. Implement an instance of IZoomVideoSDKVideoSource.
  2. Pass that implementation in externalShareSource() in IZoomVideoSDKSessionContext before joining a session.
  3. Obtain IZoomVideoSDKVideoSender from the onInitialize callback.
  4. Use sendVideoFrame in onStartSend to send raw video data.
ZoomVideoSDKVideoSource *virtual_video_source = new ZoomVideoSDKVideoSource();
session_context.externalVideoSource = virtual_video_source;
void ZoomVideoSDKVideoSource::onInitialize(IZoomVideoSDKVideoSender *sender, IVideoSDKVector<VideoSourceCapability> *support_cap_list, VideoSourceCapability &suggest_cap)
{
    this->video_sender_ = sender;
}
void ZoomVideoSDKVideoSource::onPropertyChange(IVideoSDKVector<VideoSourceCapability> *support_cap_list, VideoSourceCapability suggest_cap) {}
void ZoomVideoSDKVideoSource::onStartSend()
{
    sendVideoFrame(char *frameBuffer, int width, int height, int frameLength, int rotation)
}
void ZoomVideoSDKVideoSource::onStopSend() {}
void ZoomVideoSDKVideoSource::onUninitialized() {}

In the onInitialize(IZoomVideoSDKVideoSender *sender, IVideoSDKVector<VideoSourceCapability> *support_cap_list, VideoSourceCapability &suggest_cap) callback:

  • The support_cap_list parameter is the supported capability list. It combines the supported resolution of the session and the device to create a list of supported capabilities.
  • The suggest_cap parameter is the suggested capability. It combines the maximum capability of the session itself and the device to get the real maximum as the suggested capability.

Send raw audio data

In addition to receiving and processing raw audio data, you can also send raw or processed audio data of a user from the user's device. To do so:

  1. Implement an instance of IZoomVideoSDKVirtualAudioMic.
  2. Pass that implementation into ZoomVideoSDKSessionContext before joining a session.
    • Ensure that audioOption.mute = false; session_context.audioOption.connect = true; in ZoomVideoSDKSessionContext
  3. Obtain IZoomVideoSDKAudioSender from the onMicInitialize callback.
  4. Use send in onMicStartSend to send raw audio data.
ZoomVideoSDKVirtualAudioMic virtualMic = new ZoomVideoSDKVirtualAudioMic()
{
    session_context.virtualAudioMic = virtualMic;
    session_context.audioOption.connect = true;
    session_context.audioOption.mute = false;
    void ZoomVideoSDKVirtualAudioMic::onMicInitialize(ZOOM_VIDEO_SDK_NAMESPACE::IZoomVideoSDKAudioSender * rawdata_sender)
    {
        // Store rawdata_sender for later use
    }
    void ZoomVideoSDKVirtualAudioMic::onMicStartSend()
    {
        // You can now send audio
        rawdata_sender->send(rawData, lengthInBytes, sampleRate);
    }
    void ZoomVideoSDKVirtualAudioMic::onMicStopSend()
    {
        // You can no longer send audio
    }
    void ZoomVideoSDKVirtualAudioMic::onMicUninitialized()
    {
        // Mic is no longer active
    }
}

Send raw share data

You can also send raw screen share data. To do so:

  1. Implement an instance of IZoomVideoSDKShareSource.
  2. Pass that implementation into startSharingExternalSource() found in IZoomVideoSDKShareHelper when you are in a session.
  3. Call getShareHelper() to access IZoomVideoSDKShareHelper.
  4. Use sendShareFrame in onShareSendStarted to send raw screen share data.
ZoomVideoSDKShareSource *virtual_share_source = new ZoomVideoSDKShareSource();
ZoomVideoSDKErrors err2 = video_sdk_obj->getShareHelper()->startSharingExternalSource(virtual_share_source);
void ZoomVideoSDKShareSource::onShareSendStarted(IZoomVideoSDKShareSender *pSender){
    sendShareFrame(char *frameBuffer, int width, int height, int frameLength)};
void ZoomVideoSDKShareSource::onShareSendStopped(){};