# Raw data

<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

The Video SDK provides you with an option to access real-time raw audio, video, and share data during a session, so you can apply additional effects and enhance the user experience. Use a processor component to apply effects or transformations to each piece of data in real-time.

An example of raw share data is when a user shares their screen.

-   Receive and send raw share video data using the same logic as video data, except get the share pipe instead of the video pipe in the raw data pipe method.
-   Receive and send raw share audio data using the same method used for raw audio data.

Follow the process outlined in this flowchart to capture and process raw data.

![Flowchart outlining the raw data flow](/img/raw-data-flow.png)

This sample app demonstrates raw video and audio access in a headless Docker environment.

-   [Zoom Video SDK Linux Sample App](https://github.com/zoom/videosdk-linux-raw-recording-sample)

## Prerequisites

The Video SDK for Linux requires `pulseaudio` to enable audio raw data functions.

If you're using a Docker container to create your app, you'll need to do the following.

1. [Install pulseaudio](https://github.com/mviereck/x11docker/wiki/Container-sound:-ALSA-or-Pulseaudio) before implementing the audio raw data function.
2. Create a configuration file at `~/.config/zoomus.conf` with the following contents.

    ```ini
    [General]
    system.audio.type=default
    ```

## Select raw data memory mode

In order to utilize raw data of any type, you must first select the memory mode you wish to use. The SDK supports heap-based and stack-based memory modes.

### Stack-based memory

-   Variables are allocated automatically and deallocated after the data leaves scope.
-   Variables are not accessible from or transferable to other threads.
-   Typically has faster access than heap-based memory allocation.
-   Memory space is managed by the CPU and will not become fragmented.
-   Variables cannot be resized.

See [Stack-based memory allocation](https://en.wikipedia.org/wiki/Stack-based_memory_allocation) for details.

Specify with:

```cpp
ZoomVideoSDKRawDataMemoryModeStack
```

### Heap-based memory

-   Variables are allocated and deallocated manually. You must allocate and free variables to avoid memory leaks.
-   Variables can be accessed globally.
-   Has relatively slower access than stack-based memory allocation.
-   Has no guarantee on the efficiency of memory space and can become fragmented.
-   Variables can be resized.

See [Heap-based dynamic memory allocation](https://en.wikipedia.org/wiki/Memory_management#DYNAMIC) for details.

Specify with:

```cpp
ZoomVideoSDKRawDataMemoryModeHeap
```

## Specify memory mode

After determining which memory mode is right for you, you must specify it when the SDK is initialized. Note that this must be done for audio. video, and share individually. To specify a raw data memory mode, provide one of these enums cases to the `ZoomVideoSDKInitParams` during SDK initialization.

```cpp
ZoomVideoSDKInitParams init_params;
init_params.videoRawDataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap;
init_params.shareRawDataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap;
init_params.audioRawDataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap;
```

## Receive raw video data

Raw video data is encoded in the YUV420p format. YUV420 is a data object commonly used by the renderer based on OpenGL ES. To access and modify the video data, you must follow these steps.

1. Implement an instance of the `IZoomVideoSDKRawDataPipeDelegate`.
2. Use callback functions provided by the `IZoomVideoSDKRawDataPipeDelegate` to receive each frame of the raw video data.
3. Pass the delegate into the video pipe of a specific user.

```cpp
ZoomVideoSDKRawDataPipeDelegate *dataDelegate = new ZoomVideoSDKRawDataPipeDelegate();

void ZoomVideoSDKRawDataPipeDelegate::onRawDataFrameReceived(YUVRawDataI420 *data){
}

void ZoomVideoSDKRawDataPipeDelegate::onRawDataStatusChanged(RawDataStatus status){
}

ZoomVideoSDKRawDataPipe pipe = user.getVideoPipe();
pipe.subscribe(ZoomVideoSDKVideoResolution.VideoResolution_360P, dataDelegate);
```

Each frame of video data will be made available through the YUVRawDataI420 object. Various pieces of data can be accessed through this object in onRawDataFrameReceived:

```cpp
const int width = data->GetStreamWidth();
const int height = data->GetStreamHeight();
const int bufLen = data->GetBufferLen();
const int rotation = data->GetRotation();
const int sourceID = data->GetSourceID();
```

The Video SDK for Linux supports receiving videos in the resolutions enumerated in the reference entry for `ZoomSDKVideoResolution` on the Video SDK for Linux reference page.

## Receive raw audio data

Through your implementation of `IZoomVideoSDKDelegate`, you can access mixed (combined audio output from one or more users, as heard in a session) and per-user raw audio data.

> Unlike raw video data, raw audio data will default to stack-based memory if you do not specify a memory mode.

-   The **virtual speaker** allows access to audio data received from other users in the session. This data represents what a user would hear played through the device's speakers.
-   The **virtual mic** allows audio data for the current user to be sent to the session programmatically instead of from the SDK capturing it through an audio input device.

You can receive raw audio if it was sent through `IZoomVideoSDKVirtualAudioMic`.

To access raw audio data, you will need to:

1. Access `IZoomVideoSDKAudioHelper` using `getAudioHelper`.
2. Access `startAudio` using `IZoomVideoSDKAudioHelper`.
3. Subscribe to audio using `IZoomVideoSDKAudioHelper`.
4. Listen for the following callbacks in your listener.

```cpp
IZoomVideoSDKAudioHelper* m_pAudiohelper=  video_sdk_obj->getAudioHelper();
m_pAudiohelper->startAudio();
m_pAudiohelper->subscribe();

virtual void onMixedAudioRawDataReceived(AudioRawData *data_){
   // Access mixed data for the whole session here
};

virtual void onOneWayAudioRawDataReceived(AudioRawData *data_, IZoomVideoSDKUser *pUser){
// Access user-specific audio raw data here for the user associated with the userId
};
```

From within the callbacks, you can access the data buffer from `AudioRawData` object using `data_->getBuffer()`.

### Receive raw audio for virtual speaker

Use the virtual speaker to process audio sent to the speaker and virtual microphone to process audio received from the microphone. If you receive or send audio directly, you won't be able to process it.

Follow these steps to receive raw audio if it was sent through `IZoomVideoSDKVirtualAudioMic`.

1. Implement an instance of `IZoomVideoSDKVirtualAudioSpeaker`.
2. Pass that implementation into `ZoomVideoSDKSessionContext`. Note that this is done before joining a session.
3. Access raw data in each callback method.

```cpp
 ZoomVideoSDKVirtualAudioSpeaker* vSpeaker  =new ZoomVideoSDKVirtualAudioSpeaker();
        session_context.virtualAudioSpeaker =vSpeaker;

virtual void onVirtualSpeakerMixedAudioReceived(AudioRawData* data_){
    printf("onVirtualSpeakerMixedAudioReceived() main \n");
    printf("data %s \n",  data_->GetBuffer());
}
```

## Receive raw share data

Receive raw share video and audio data, for example, when someone is sharing their screen, similarly to how you receive raw video and raw audio data.

### Receive raw share video data

Follow the same steps and code to [receive raw video data](#receive-raw-video-data), except get the share data instead of the video data, see the following code for an example.

```cpp
// Previous code same as receive raw video

ZoomVideoSDKRawDataPipeDelegate *dataDelegate = new ZoomVideoSDKRawDataPipeDelegate();

void ZoomVideoSDKRawDataPipeDelegate::onRawDataFrameReceived(YUVRawDataI420 *data){
}

void ZoomVideoSDKRawDataPipeDelegate::onRawDataStatusChanged(RawDataStatus status){
}

ZoomVideoSDKRawDataPipe pipe = user.GetSharePipe();
pipe.subscribe(ZoomVideoSDKVideoResolution.VideoResolution_360P, dataDelegate);
```

### Receive raw share audio data

Receive raw share audio data in the same way as you [receive raw audio data](#receive-raw-audio-data).

## Send raw video data

Follow these steps to send raw as well as processed video data of a user from the user's device.

1. Implement an instance `IZoomVideoSDKVideoSource`.
2. Pass that implementation in `externalShareSource()` found in the `IZoomVideoSDKSessionContext` object before joining a session.
3. Obtain `IZoomVideoSDKVideoSender` from the `onInitialize` callback.
4. Use the `sendVideoFrame` method in `onStartSend` to send raw video data.

```cpp
ZoomVideoSDKVideoSource* virtual_video_source = new ZoomVideoSDKVideoSource();
    session_context.externalVideoSource=virtual_video_source;

    void ZoomVideoSDKVideoSource::onInitialize(IZoomVideoSDKVideoSender* sender, IVideoSDKVector<VideoSourceCapability >* support_cap_list, VideoSourceCapability& suggest_cap) {
        this->video_sender_=sender;
    }

    void ZoomVideoSDKVideoSource::onPropertyChange(IVideoSDKVector<VideoSourceCapability >* support_cap_list, VideoSourceCapability suggest_cap) {}

    void ZoomVideoSDKVideoSource::onStartSend() {
        sendVideoFrame(char* frameBuffer, int width, int height, int frameLength, int rotation)
    }

    void ZoomVideoSDKVideoSource::onStopSend()  {}

    void ZoomVideoSDKVideoSource:: onUninitialized() {}
```

In the `onInitialize(IZoomVideoSDKVideoSender *sender, IVideoSDKVector<VideoSourceCapability> *support_cap_list, VideoSourceCapability &suggest_cap)` callback:

-   The `support_cap_list` parameter is the supported capability list. It combines the supported resolution of the session and the device to create a list of supported capabilities.
-   The `suggest_cap` parameter is the suggested capability. It combines the maximum capability of the session itself and the device to get the real maximum as the suggested capability.

## Send raw audio data

Follow these steps to send raw or processed audio data of a user from the user's device.

1. Implement an instance of `IZoomVideoSDKVirtualAudioMic`.
2. Pass that implementation into the `ZoomVideoSDKSessionContext` object before joining a session.
    - Ensure that` audioOption.mute = false;` and `session_context.audioOption.connect = true;` in `ZoomVideoSDKSessionContext`. 3. Obtain `IZoomVideoSDKAudioSender` from the `onMicInitialize` callback.
3. Use the send method in `onMicStartSend` to send raw audio data.

```cpp
ZoomVideoSDKVirtualAudioMic virtualMic = new ZoomVideoSDKVirtualAudioMic() {
  session_context.virtualAudioMic=virtualMic;
  session_context.audioOption.connect = true;
  session_context.audioOption.mute = false;
}

void ZoomVideoSDKVirtualAudioMic::onMicInitialize(ZOOM_VIDEO_SDK_NAMESPACE::IZoomVideoSDKAudioSender* rawdata_sender){
    // Store rawdata_sender for later use
}

void ZoomVideoSDKVirtualAudioMic::onMicStartSend(){
    // You can now send audio
    rawdata_sender->send(rawData, lengthInBytes, sampleRate);
}

void ZoomVideoSDKVirtualAudioMic::onMicStopSend(){
    // You can no longer send audio
}

void ZoomVideoSDKVirtualAudioMic::onMicUninitialized(){
    // Mic is no longer active
}
```

## Send raw share data

Follow these steps to send raw screen share data.

1. Implement an instance of `IZoomVideoSDKShareSource`.
2. Pass that implementation into `startSharingExternalSource()` found in the `IZoomVideoSDKShareHelper` object when you are in a session.
    - Call `getShareHelper()` to access `IZoomVideoSDKShareHelper`.
3. Use the `sendShareFrame` method in `onShareSendStarted` to send raw screen share data.

```cpp
 ZoomVideoSDKShareSource* virtual_share_source = new ZoomVideoSDKShareSource();
    ZoomVideoSDKErrors err2= video_sdk_obj->getShareHelper()->startSharingExternalSource(virtual_share_source);

    void ZoomVideoSDKShareSource::onShareSendStarted(IZoomVideoSDKShareSender* pSender) {
        sendShareFrame(char* frameBuffer, int width, int height, int frameLength)
    };

    void ZoomVideoSDKShareSource::onShareSendStopped() {
    };
```

## Callbacks

Within callbacks, you can access the data buffer with `rawData_->GetBuffer()`. See **Integrate** for more details on implementing your `IZoomVideoSDKDelegate`. See the following sections for examples.

### Get notified when shared raw audio data is received

```cpp
virtual void onSharedAudioRawDataReceived(AudioRawData* data_) {
}
```

### Get notified when mixed raw audio data is received

```cpp
virtual void onMixedAudioRawDataReceived(AudioRawData* data_) {
}
```

### Get notified when one-way raw audio data is received

```cpp
virtual void onOneWayAudioRawDataReceived(AudioRawData* data_, IZoomVideoSDKUser* pUser) {
}
```

## From the developer blog

See the following blog post for a walkthrough.

-   [How to get raw streaming video and audio from Video SDK web sessions using Linux](/blog/video-sdk-raw-streaming-linux) by Chun Siong Tan - 03-04-2024