Receive raw data

There are two ways to consume incoming media as raw data:

Video and share frames: subscribe a ZoomVideoSDKRawDataPipeDelegate to a user's video or share pipe.
Audio frames: install a ZoomVideoSDKVirtualAudioSpeaker on the session context at join time, or listen for the audio raw-data callbacks on your ZoomVideoSDKDelegate.

Receive raw video frames

Raw video arrives as YUV (I420) data: three planes (Y, U, V) plus dimensions and rotation. Implement ZoomVideoSDKRawDataPipeDelegate and pass it to a user's video pipe.

val dataDelegate = object : ZoomVideoSDKRawDataPipeDelegate {
    override fun onRawDataStatusChanged(status: ZoomVideoSDKRawDataPipeDelegate.RawDataStatus?) {
        // The pipe is now active or inactive.
    }
    override fun onRawDataFrameReceived(rawData: ZoomVideoSDKVideoRawData?) {
        rawData ?: return
        // Read YUV planes and dimensions — see "Read a video frame" below.
    }
}
val pipe = user.videoPipe
val result = pipe.subscribe(
    ZoomVideoSDKVideoResolution.VideoResolution_360P,
    dataDelegate
)
if (result != ZoomVideoSDKErrors.Errors_Success) {
    // Subscription was rejected.
}

ZoomVideoSDKRawDataPipeDelegate dataDelegate = new ZoomVideoSDKRawDataPipeDelegate() {
    @Override
    public void onRawDataStatusChanged(ZoomVideoSDKRawDataPipeDelegate.RawDataStatus status) {
        // The pipe is now active or inactive.
    }
    @Override
    public void onRawDataFrameReceived(ZoomVideoSDKVideoRawData rawData) {
        if (rawData == null) return;
        // Read YUV planes and dimensions — see "Read a video frame" below.
    }
};
ZoomVideoSDKRawDataPipe pipe = user.getVideoPipe();
int result = pipe.subscribe(
    ZoomVideoSDKVideoResolution.VideoResolution_360P,
    dataDelegate
);
if (result != ZoomVideoSDKErrors.Errors_Success) {
    // Subscription was rejected.
}

subscribe returns an int: ZoomVideoSDKErrors.Errors_Success (value 0) means frames will start arriving on onRawDataFrameReceived; any other value is an error code from ZoomVideoSDKErrors.

Read a video frame

Inside onRawDataFrameReceived, read planes and metadata from the ZoomVideoSDKVideoRawData.

val width = rawData.streamWidth
val height = rawData.streamHeight
val yBuffer = rawData.yBuffer  // ByteBuffer for the Y plane
val uBuffer = rawData.uBuffer  // ByteBuffer for the U plane
val vBuffer = rawData.vBuffer  // ByteBuffer for the V plane
val rotation = rawData.rotation   // 0, 90, 180, or 270 (clockwise)
val format = rawData.format       // currently always FORMAT_TYPE_I420
val timestamp = rawData.timeStamp

int width = rawData.getStreamWidth();
int height = rawData.getStreamHeight();
ByteBuffer yBuffer = rawData.getyBuffer();  // Y plane
ByteBuffer uBuffer = rawData.getuBuffer();  // U plane
ByteBuffer vBuffer = rawData.getvBuffer();  // V plane
int rotation = rawData.getRotation();       // 0, 90, 180, or 270 (clockwise)
int format = rawData.getFormat();           // currently always FORMAT_TYPE_I420
long timestamp = rawData.getTimeStamp();

The getter casing (getyBuffer, getuBuffer, getvBuffer) reflects the SDK method names. Don't "correct" the lowercase letter when grepping or wiring up reflection.

Retain a frame past the callback

Raw data buffers are valid only inside onRawDataFrameReceived by default. Once the callback returns, the buffer may be recycled. To hold onto a frame (for example, to hand it off to a background thread for processing or encoding), use the SDK's reference counting.

override fun onRawDataFrameReceived(rawData: ZoomVideoSDKVideoRawData?) {
    rawData ?: return
    if (rawData.canAddRef()) {
        rawData.addRef()
        backgroundExecutor.execute {
            try {
                processFrame(rawData)
            } finally {
                rawData.releaseRef()
            }
        }
    }
}

@Override
public void onRawDataFrameReceived(ZoomVideoSDKVideoRawData rawData) {
    if (rawData == null) return;
    if (rawData.canAddRef()) {
        rawData.addRef();
        backgroundExecutor.execute(() -> {
            try {
                processFrame(rawData);
            } finally {
                rawData.releaseRef();
            }
        });
    }
}

Always pair addRef() with releaseRef(). Orphaned refs leak frame buffers.

Stop receiving frames

Call unSubscribe(delegate) on the same pipe when you no longer want frames. Always unsubscribe before tearing down the Activity, otherwise the SDK keeps invoking your delegate against a destroyed context.

user.videoPipe.unSubscribe(dataDelegate)

user.getVideoPipe().unSubscribe(dataDelegate);

Receive raw audio frames

Through your ZoomVideoSDKDelegate, you can listen for mixed (combined audio from one or more users), per-user, and share audio raw data callbacks. The callbacks fire continuously while any audio is flowing in the session.

override fun onMixedAudioRawDataReceived(rawData: ZoomVideoSDKAudioRawData?) {
    // Combined audio for the whole session.
}
override fun onOneWayAudioRawDataReceived(
    rawData: ZoomVideoSDKAudioRawData?,
    user: ZoomVideoSDKUser
) {
    // Per-user audio.
}
override fun onShareAudioRawDataReceived(rawData: ZoomVideoSDKAudioRawData?) {
    // Audio from an active screen share.
}

@Override
public void onMixedAudioRawDataReceived(ZoomVideoSDKAudioRawData rawData) {
    // Combined audio for the whole session.
}
@Override
public void onOneWayAudioRawDataReceived(
    ZoomVideoSDKAudioRawData rawData,
    ZoomVideoSDKUser user
) {
    // Per-user audio.
}
@Override
public void onShareAudioRawDataReceived(ZoomVideoSDKAudioRawData rawData) {
    // Audio from an active screen share.
}

Inside each callback, the ZoomVideoSDKAudioRawData exposes the PCM buffer through rawData.buffer (Kotlin) / rawData.getBuffer() (Java).

Receive audio through a virtual speaker

You can also install a ZoomVideoSDKVirtualAudioSpeaker on the session context when joining. This routes received audio to your speaker implementation instead of (or in addition to) the device's audio output. Use this when you want to process incoming audio without playing it through the speaker.

val virtualSpeaker = object : ZoomVideoSDKVirtualAudioSpeaker {
    override fun onVirtualSpeakerMixedAudioReceived(rawData: ZoomVideoSDKAudioRawData?) {
        // Combined session audio.
    }
    override fun onVirtualSpeakerOneWayAudioReceived(
        rawData: ZoomVideoSDKAudioRawData?,
        user: ZoomVideoSDKUser?
    ) {
        // Per-user audio.
    }
    override fun onVirtualSpeakerShareAudioReceived(rawData: ZoomVideoSDKAudioRawData?) {
        // Share audio.
    }
}
val context = ZoomVideoSDKSessionContext().apply {
    virtualAudioSpeaker = virtualSpeaker
}

ZoomVideoSDKVirtualAudioSpeaker virtualSpeaker = new ZoomVideoSDKVirtualAudioSpeaker() {
    @Override
    public void onVirtualSpeakerMixedAudioReceived(ZoomVideoSDKAudioRawData rawData) {
        // Combined session audio.
    }
    @Override
    public void onVirtualSpeakerOneWayAudioReceived(
        ZoomVideoSDKAudioRawData rawData,
        ZoomVideoSDKUser user
    ) {
        // Per-user audio.
    }
    @Override
    public void onVirtualSpeakerShareAudioReceived(ZoomVideoSDKAudioRawData rawData) {
        // Share audio.
    }
};
ZoomVideoSDKSessionContext context = new ZoomVideoSDKSessionContext();
context.virtualAudioSpeaker = virtualSpeaker;

Receive raw share data

A screen share streams as both video frames (the screen capture) and optionally audio (the device audio the sharer included with enableShareDeviceAudio).

Share video

Get the share pipe from the sharing user, then subscribe exactly like you would for video.

val pipe = user.sharePipe
pipe.subscribe(
    ZoomVideoSDKVideoResolution.VideoResolution_360P,
    dataDelegate
)

ZoomVideoSDKRawDataPipe pipe = user.getSharePipe();
pipe.subscribe(
    ZoomVideoSDKVideoResolution.VideoResolution_360P,
    dataDelegate
);

The same onRawDataFrameReceived callback fires for share frames; the same accessors and lifecycle rules apply.

Share audio

Listen for onShareAudioRawDataReceived on your ZoomVideoSDKDelegate (see Receive raw audio frames above) or onVirtualSpeakerShareAudioReceived on a ZoomVideoSDKVirtualAudioSpeaker. The buffer format and access pattern is the same as session audio.