How to Use OpenCV with ZED in C++

Introduction

In this tutorial, you will learn how to to capture and display color and depth images using OpenCV and the ZED SDK in C++.

Code Example

Sample code is available on GitHub.

The ZED SDK provides its own sl::Mat class to store image data, with a structure and data types similar to OpenCV cv::Mat. Image data can be shared between sl::Mat and cv::Mat by having the both memory pointers pointing to the same address. Therefore, no data transfer is required between the two matrices.

To convert an sl::Mat to a cv::Mat, we provide an slMat2cvMat() function in the OpenCV sample.

// Create a sl::Mat object (4 channels of type unsigned char) to store the image.
sl::Mat image_zed(zed.getResolution(), MAT_TYPE::U8_C4);
// Create an OpenCV Mat that shares sl::Mat data
cv::Mat image_ocv = slMat2cvMat(image_zed);

Capturing Video

To capture video, use grab() and retrieveImage(). Since sl::Mat and cv::Mat share the same data, calling retrieveImage() will update the OpenCV matrix as well. Display the video with OpenCV using cv::imshow().

if (zed.grab() == ERROR_CODE::SUCCESS) {
    // Retrieve the left image in sl::Mat
    // The cv::Mat is automatically updated
    zed.retrieveImage(image_zed, VIEW::LEFT);
    // Display the left image from the cv::Mat object
    cv::imshow("Image", image_ocv);
}

Capturing Depth

A depth map is a 1-channel matrix with 32-bit float values for each pixel. Each value expresses the distance of a pixel in the scene. The depth map can be retrieved using retrieveMeasure() and shared with a cv::Mat. Please refer to the Depth API for more information.

// Create a sl::Mat with float type (32-bit)
sl::Mat depth_zed(zed.getResolution(), MAT_TYPE::F32_C1);
// Create an OpenCV Mat that shares sl::Mat data
cv::Mat depth_ocv = slMat2cvMat(depth_zed);

if (zed.grab() == ERROR_CODE::SUCCESS) {
    // Retrieve the depth measure (32-bit)
    zed.retrieveMeasure(depth_zed, MEASURE::DEPTH);
    // Print the depth value at the center of the image
    std::cout << depth_ocv.at<float>(depth_ocv.rows/2, depth_ocv.cols/2) << std::endl;
}

Displaying Depth

A cv::Mat with 32-bit float values can’t be displayed with cv::imshow. If you want to display the depth map you have to normalize it to fit in an unsigned char matrix. A call to sl::Camera::retrieveImage with VIEW::DEPTH returns such an image of the depth data as a grayscale image. Do not use this representation for other purposes than displaying the image.

// Create a sl::Mat object (4 channels of type unsigned char) to store the image.
sl::Mat image_depth_zed(zed.getResolution(), MAT_TYPE::U8_C4);
// Create an OpenCV Mat that shares sl::Mat data
cv::Mat image_depth_ocv = slMat2cvMat(image_depth_zed);

if (zed.grab() == ERROR_CODE::SUCCESS) {
    // Retrieve the normalized depth image
    zed.retrieveImage(image_depth_zed, VIEW::DEPTH);
    // Display the depth view from the cv::Mat object
    cv::imshow("Image", image_depth_ocv);  
}

UVC Capture

You can also use the ZED as a standard UVC camera in OpenCV to capture raw stereo video using the code snippet below. To get rectified images and calibration with OpenCV, use the native capture sample available on GitHub.

// Open the ZED camera
VideoCapture cap(0);
if(!cap.isOpened())
    return -1;

// Set the video resolution to HD720 (2560*720)
cap.set(CV_CAP_PROP_FRAME_WIDTH, 2560);
cap.set(CV_CAP_PROP_FRAME_HEIGHT, 720);

for(;;)
{
    Mat frame, left, right;
    // Get a new frame from camera
    cap >> frame;
    // Extract left and right images from side-by-side
    left_image = frame(cv::Rect(0, 0, frame.cols / 2, frame.rows));
    right_image = frame(cv::Rect(frame.cols / 2, 0, frame.cols / 2, frame.rows));
    // Display images    
    imshow("frame", frame);
    imshow("left", left_image);
    imshow("right", right_image);
    if(waitKey(30) >= 0) break;
}
// Deinitialize camera in the VideoCapture destructor
return 0;