Raw data

The Video SDK provides you with an option to access real-time raw audio, video, and share data during a session, so you can apply additional effects and enhance the user experience. Use a processor component to apply effects or transformations to each piece of data in real-time.

An example of raw share data is when a user shares their screen.

  • Receive and send raw share video data using the same logic as video data, except get the share pipe instead of the video pipe in the raw data pipe method.
  • Receive and send raw share audio data using the same method used for raw audio data.

Follow the process outlined in this flowchart to capture and process raw data.

Flowchart outlining the raw data flow

This sample app demonstrates raw video and audio access in a headless Docker environment.

Prerequisites

The Video SDK for Linux requires pulseaudio to enable audio raw data functions.

If you're using a Docker container to create your app, you'll need to do the following.

  1. Install pulseaudio before implementing the audio raw data function.

  2. Create a configuration file at ~/.config/zoomus.conf with the following contents.

    [General]
system.audio.type=default

Select raw data memory mode

In order to utilize raw data of any type, you must first select the memory mode you wish to use. The SDK supports heap-based and stack-based memory modes.

Stack-based memory

  • Variables are allocated automatically and deallocated after the data leaves scope.
  • Variables are not accessible from or transferable to other threads.
  • Typically has faster access than heap-based memory allocation.
  • Memory space is managed by the CPU and will not become fragmented.
  • Variables cannot be resized.

See Stack-based memory allocation for details.

Specify with:

ZoomVideoSDKRawDataMemoryModeStack

Heap-based memory

  • Variables are allocated and deallocated manually. You must allocate and free variables to avoid memory leaks.
  • Variables can be accessed globally.
  • Has relatively slower access than stack-based memory allocation.
  • Has no guarantee on the efficiency of memory space and can become fragmented.
  • Variables can be resized.

See Heap-based dynamic memory allocation for details.

Specify with:

ZoomVideoSDKRawDataMemoryModeHeap

Specify memory mode

After determining which memory mode is right for you, you must specify it when the SDK is initialized. Note that this must be done for audio. video, and share individually. To specify a raw data memory mode, provide one of these enums cases to the ZoomVideoSDKInitParams during SDK initialization.

ZoomVideoSDKInitParams init_params;
init_params.videoRawDataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap;
init_params.shareRawDataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap;
init_params.audioRawDataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap;

Receive raw video data

Raw video data is encoded in the YUV420p format. YUV420 is a data object commonly used by the renderer based on OpenGL ES. To access and modify the video data, you must follow these steps.

  1. Implement an instance of the IZoomVideoSDKRawDataPipeDelegate.
  2. Use callback functions provided by the IZoomVideoSDKRawDataPipeDelegate to receive each frame of the raw video data.
  3. Pass the delegate into the video pipe of a specific user.
ZoomVideoSDKRawDataPipeDelegate *dataDelegate = new ZoomVideoSDKRawDataPipeDelegate();
void ZoomVideoSDKRawDataPipeDelegate::onRawDataFrameReceived(YUVRawDataI420 *data){
}
void ZoomVideoSDKRawDataPipeDelegate::onRawDataStatusChanged(RawDataStatus status){
}
ZoomVideoSDKRawDataPipe pipe = user.getVideoPipe();
pipe.subscribe(ZoomVideoSDKVideoResolution.VideoResolution_360P, dataDelegate);

Each frame of video data will be made available through the YUVRawDataI420 object. Various pieces of data can be accessed through this object in onRawDataFrameReceived:

const int width = data->GetStreamWidth();
const int height = data->GetStreamHeight();
const int bufLen = data->GetBufferLen();
const int rotation = data->GetRotation();
const int sourceID = data->GetSourceID();

The Video SDK for Linux supports receiving videos in the resolutions enumerated in the reference entry for ZoomSDKVideoResolution on the Video SDK for Linux reference page.

Receive raw audio data

Through your implementation of IZoomVideoSDKDelegate, you can access mixed (combined audio output from one or more users, as heard in a session) and per-user raw audio data.

Unlike raw video data, raw audio data will default to stack-based memory if you do not specify a memory mode.

  • The virtual speaker allows access to audio data received from other users in the session. This data represents what a user would hear played through the device's speakers.
  • The virtual mic allows audio data for the current user to be sent to the session programmatically instead of from the SDK capturing it through an audio input device.

You can receive raw audio if it was sent through IZoomVideoSDKVirtualAudioMic.

To access raw audio data, you will need to:

  1. Access IZoomVideoSDKAudioHelper using getAudioHelper.
  2. Access startAudio using IZoomVideoSDKAudioHelper.
  3. Subscribe to audio using IZoomVideoSDKAudioHelper.
  4. Listen for the following callbacks in your listener.
IZoomVideoSDKAudioHelper* m_pAudiohelper=  video_sdk_obj->getAudioHelper();
m_pAudiohelper->startAudio();
m_pAudiohelper->subscribe();
virtual void onMixedAudioRawDataReceived(AudioRawData *data_){
   // Access mixed data for the whole session here
};
virtual void onOneWayAudioRawDataReceived(AudioRawData *data_, IZoomVideoSDKUser *pUser){
// Access user-specific audio raw data here for the user associated with the userId
};

From within the callbacks, you can access the data buffer from AudioRawData object using data_->getBuffer().

Receive raw audio for virtual speaker

Use the virtual speaker to process audio sent to the speaker and virtual microphone to process audio received from the microphone. If you receive or send audio directly, you won't be able to process it.

Follow these steps to receive raw audio if it was sent through IZoomVideoSDKVirtualAudioMic.

  1. Implement an instance of IZoomVideoSDKVirtualAudioSpeaker.
  2. Pass that implementation into ZoomVideoSDKSessionContext. Note that this is done before joining a session.
  3. Access raw data in each callback method.
 ZoomVideoSDKVirtualAudioSpeaker* vSpeaker  =new ZoomVideoSDKVirtualAudioSpeaker();
        session_context.virtualAudioSpeaker =vSpeaker;
virtual void onVirtualSpeakerMixedAudioReceived(AudioRawData* data_){
    printf("onVirtualSpeakerMixedAudioReceived() main \n");
    printf("data %s \n",  data_->GetBuffer());
}

Receive raw share data

Receive raw share video and audio data, for example, when someone is sharing their screen, similarly to how you receive raw video and raw audio data.

Receive raw share video data

Follow the same steps and code to receive raw video data, except get the share data instead of the video data, see the following code for an example.

// Previous code same as receive raw video
ZoomVideoSDKRawDataPipeDelegate *dataDelegate = new ZoomVideoSDKRawDataPipeDelegate();
void ZoomVideoSDKRawDataPipeDelegate::onRawDataFrameReceived(YUVRawDataI420 *data){
}
void ZoomVideoSDKRawDataPipeDelegate::onRawDataStatusChanged(RawDataStatus status){
}
ZoomVideoSDKRawDataPipe pipe = user.GetSharePipe();
pipe.subscribe(ZoomVideoSDKVideoResolution.VideoResolution_360P, dataDelegate);

Receive raw share audio data

Receive raw share audio data in the same way as you receive raw audio data.

Send raw video data

Follow these steps to send raw as well as processed video data of a user from the user's device.

  1. Implement an instance IZoomVideoSDKVideoSource.
  2. Pass that implementation in externalShareSource() found in the IZoomVideoSDKSessionContext object before joining a session.
  3. Obtain IZoomVideoSDKVideoSender from the onInitialize callback.
  4. Use the sendVideoFrame method in onStartSend to send raw video data.
ZoomVideoSDKVideoSource* virtual_video_source = new ZoomVideoSDKVideoSource();
    session_context.externalVideoSource=virtual_video_source;
    void ZoomVideoSDKVideoSource::onInitialize(IZoomVideoSDKVideoSender* sender, IVideoSDKVector<VideoSourceCapability >* support_cap_list, VideoSourceCapability& suggest_cap) {
        this->video_sender_=sender;
    }
    void ZoomVideoSDKVideoSource::onPropertyChange(IVideoSDKVector<VideoSourceCapability >* support_cap_list, VideoSourceCapability suggest_cap) {}
    void ZoomVideoSDKVideoSource::onStartSend() {
        sendVideoFrame(char* frameBuffer, int width, int height, int frameLength, int rotation)
    }
    void ZoomVideoSDKVideoSource::onStopSend()  {}
    void ZoomVideoSDKVideoSource:: onUninitialized() {}

In the onInitialize(IZoomVideoSDKVideoSender *sender, IVideoSDKVector<VideoSourceCapability> *support_cap_list, VideoSourceCapability &suggest_cap) callback:

  • The support_cap_list parameter is the supported capability list. It combines the supported resolution of the session and the device to create a list of supported capabilities.
  • The suggest_cap parameter is the suggested capability. It combines the maximum capability of the session itself and the device to get the real maximum as the suggested capability.

Send raw audio data

Follow these steps to send raw or processed audio data of a user from the user's device.

  1. Implement an instance of IZoomVideoSDKVirtualAudioMic.
  2. Pass that implementation into the ZoomVideoSDKSessionContext object before joining a session.
    • Ensure that audioOption.mute = false; and session_context.audioOption.connect = true; in ZoomVideoSDKSessionContext. 3. Obtain IZoomVideoSDKAudioSender from the onMicInitialize callback.
  3. Use the send method in onMicStartSend to send raw audio data.
ZoomVideoSDKVirtualAudioMic virtualMic = new ZoomVideoSDKVirtualAudioMic() {
  session_context.virtualAudioMic=virtualMic;
  session_context.audioOption.connect = true;
  session_context.audioOption.mute = false;
}
void ZoomVideoSDKVirtualAudioMic::onMicInitialize(ZOOM_VIDEO_SDK_NAMESPACE::IZoomVideoSDKAudioSender* rawdata_sender){
    // Store rawdata_sender for later use
}
void ZoomVideoSDKVirtualAudioMic::onMicStartSend(){
    // You can now send audio
    rawdata_sender->send(rawData, lengthInBytes, sampleRate);
}
void ZoomVideoSDKVirtualAudioMic::onMicStopSend(){
    // You can no longer send audio
}
void ZoomVideoSDKVirtualAudioMic::onMicUninitialized(){
    // Mic is no longer active
}

Send raw share data

Follow these steps to send raw screen share data.

  1. Implement an instance of IZoomVideoSDKShareSource.
  2. Pass that implementation into startSharingExternalSource() found in the IZoomVideoSDKShareHelper object when you are in a session.
    • Call getShareHelper() to access IZoomVideoSDKShareHelper.
  3. Use the sendShareFrame method in onShareSendStarted to send raw screen share data.
 ZoomVideoSDKShareSource* virtual_share_source = new ZoomVideoSDKShareSource();
    ZoomVideoSDKErrors err2= video_sdk_obj->getShareHelper()->startSharingExternalSource(virtual_share_source);
    void ZoomVideoSDKShareSource::onShareSendStarted(IZoomVideoSDKShareSender* pSender) {
        sendShareFrame(char* frameBuffer, int width, int height, int frameLength)
    };
    void ZoomVideoSDKShareSource::onShareSendStopped() {
    };

Callbacks

Within callbacks, you can access the data buffer with rawData_->GetBuffer(). See Integrate for more details on implementing your IZoomVideoSDKDelegate. See the following sections for examples.

Get notified when shared raw audio data is received

virtual void onSharedAudioRawDataReceived(AudioRawData* data_) {
}

Get notified when mixed raw audio data is received

virtual void onMixedAudioRawDataReceived(AudioRawData* data_) {
}

Get notified when one-way raw audio data is received

virtual void onOneWayAudioRawDataReceived(AudioRawData* data_, IZoomVideoSDKUser* pUser) {
}

From the developer blog

See the following blog post for a walkthrough.