Getting Started with OpenNI2 and ZED

OpenNI provides a uniform interface that third party middleware developers can use to interact with depth sensors. The ZED OpenNI driver transforms each ZED camera into a PrimeSense compatible depth sensors to use it with the OpenNI 2 API.

Key Features #

Color stream for left or right camera
Depth stream relative to left or right camera
Gray stream for left or right camera
Configuration of the camera parameters with YAML files

Installation #

The latest version of OpenNI2 is required. It can be downloaded as a precompiled library from the Occipital website. The ZED OpenNI2 driver supports Windows and Linux distributions.

Linux #

Prerequisites #

CMake [v3.10+]
OpenCV [v3.4+] optionally required to build all the examples.
OpenNI2 [v2.2.0]

Install OpenNI2 #

Download the library for the correct architecture of your host system: x86, x64, ARM

📌 Note: For ARM64 devices the library must be compiled from sources.

Extract the library:

$ tar -xf OpenNI-Linux-<arch>-2.2.0.33.tar.bz2

A new folder OpenNI-Linux-<arch>-2.2 will be created containing the precompiled library, take note of its full path because you will need for the next steps.

Build the driver #

[Optional] Install OpenCV:

$ sudo apt install libopencv-dev

Clone the ZED OpenNI2 repository:

    $ git clone git@github.com:stereolabs/zed-openni.git

Build the driver (use the path to the OpenNI2 library noted above):

$ cd zed-openni
$ mkdir build
$ cd build
$ cmake .. -DOPENNI2_DIR='path-to-your-openni2-installation-folder' -DCMAKE_BUILD_TYPE=Release
$ make -j$(nproc)

The library file of the driver will be automatically copied in the folder <openni2-path>/Redist/OpenNI2/Drivers and it will be automatically loaded by OpenNI2 during initialization.

Please be sure to have write permission for the folder <openni2-path>/Redist/OpenNI2/Drivers.

Windows #

Prerequisites #

CMake [v3.10+] is required to configure the ZED OpenNI driver before building. You can install the latest version downloading it from the CMake website.
OpenNI2 [v2.2.0]

Install OpenNI2 #

Download the library for the correct architecture of your host system: x86, x64

📌 Note: For ARM devices the library must be compiled from sources.

Extract the library using the tool you prefer. A new folder OpenNI-Windows-<arch>-2.2.0.33 will be created containing the installer. Enter into the new folder and double click on the file OpenNI-Windows-<arch>-2.2.msi to install OpenNI2. Take note of the installation path because you will need it for the next steps.

📌 Note: Install the Primesense driver when asked to create the correct path for the OpenNI drivers.

Build the driver #

Clone the ZED OpenNI2 repository using Git for Windows.

Run CMake (cmake-gui), select the folder where you cloned the repository as source and setup a build folder.

Configure the following parameters

OpenNI2-DIR -> use the path to the OpenNI2 library noted above (e.g. C:\Program Files\OpenNI2)

Click on Configure, select the build environment and then click on Generate. If everything is correctly configured you can now click Open Project and then build the driver using Visual Studio.

The build process generates a file named libzeddriver.dll in the sub-folder Release of the folder chosen as build destination. This file must be manually copied in the folder of OpenNI2 containing all the available drivers to be able to run the OpenNI2 examples using the ZED camera.

Camera configuration #

The ZED camera can be initialized using two configuration files:

ZedInitConfig.yaml: contains the Initialization parameters in YAML format
ZedRuntimeConfig.yaml: contains the Runtime parameters in YAML format

Both the files, if not existing, will be automatically created with the default parameters by the driver in the same folder of the executable.

Initialization parameters #

Parameter	Description
`camera_resolution`	Define the chosen camera resolution. Small resolutions offer higher framerate and lower computation time. In most situations, the `RESOLUTION::HD720` at 60 fps is the best balance between image quality and framerate. Available resolutions are listed here: RESOLUTION.
`camera_fps`	Requested camera frame rate. If set to 0, the highest FPS of the specified `camera_resolution` will be used. See RESOLUTION for a list of supported framerates.
`camera_image_flip`	If you are using the camera upside down, setting this parameter to true will cancel its rotation. The images will be horizontally flipped.
`camera_disable_self_calib`	At initialization, the Camera runs a self-calibration process that corrects small offsets from the device’s factory calibration. A drawback is that calibration parameters will slightly change from one run to another, which can be an issue for repeatability. If set to true, self-calibration will be disabled and calibration parameters won’t be optimized.
`enable_right_side_measure`	By default, the SDK only computes a single depth map, aligned with the left camera image. This parameter allows you to enable the `MEASURE::DEPTH_RIGHT` and other `MEASURE::<XXX>_RIGHT` at the cost of additional computation time. For example, mixed reality pass-through applications require one depth map per eye, so this parameter can be activated.
`depth_mode`	The SDK offers several DEPTH_MODE options offering various levels of performance and accuracy. This parameter allows you to set the DEPTH_MODE that best matches your needs.
`depth_stabilization`	Regions of the generated depth map can oscillate from one frame to another. These oscillations result from a lack of texture (too homogeneous) on an object and by image noise. This parameter enables a stabilization filter that reduces these oscillations.
`depth_minimum_distance`	This parameter allows you to specify the minimum depth value (from the camera) that will be computed (units forced to millimeters). In stereovision (the depth technology used by the camera), looking for closer depth values can have a slight impact on performance and memory consumption. On most of modern GPUs, performance impact will be low. However, the impact of memory footprint will be visible. In cases of limited computation power, increasing this value can provide better performance.
`depth_maximum_distance`	Defines the current maximum distance that can be computed (units forced to millimeters) [max value forced to 10m because of OpenNI2 limitations]
`coordinate_units`	forced to millimeters
`coordinate_system`	forced to `COORDINATE_SYSTEM::IMAGE`
`sdk_verbose`	set SDK and driver to verbose, showing useful debug information
`enable_image_enhancement`	Enable or Disable the Enhanced Contrast Technology, to improve image quality

Runtime parameters #

Parameter	Description
`confidence_threshold`	Threshold to reject depth values based on their confidence. Each depth pixel has a corresponding confidence. (`MEASURE::CONFIDENCE`), the confidence range is `[1,100]`. By default, the confidence threshold is set at 100, meaning that no depth pixel will be rejected. Decreasing this value will remove depth data from both objects edges and low textured areas, to keep only confident depth estimation data.
`texture_confidence_threshold`	Threshold to reject depth values based on their texture confidence. The texture confidence range is `[1,100]`. By default, the texture confidence threshold is set at 100, meaning that no depth pixel will be rejected. Decreasing this value will remove depth data from image areas which are uniform.
`sensing_mode`	Defines the algorithm used for depth map computation, more info : SENSING_MODE definition.
`measure3D_reference_frame`	always forced to `sl::REFERENCE_FRAME::CAMERA`
`enable_depth`	always forced to `true`

Examples #

Windows examples #

The examples provided with this repository cannot be built for Windows because of the high number of dependencies to be satisfied that require a highly laborious and sophisticated process. You can however run the examples provided with OpenNI2. OpenNI2 must be able to recognize the ZED camera as a Primesense compatible device, this is possible manually copying the file libzeddriver.dll in the folder <path-to-your-openni2-installation-folder\Samples\Bin\OpenNI2\Drivers> and <path-to-your-openni2-installation-folder\Tools\OpenNI2\Drivers> (e.g. C:\Program Files\OpenNI2\Samples\Bin\OpenNI2\Drivers, C:\Program Files\OpenNI2\Tools\OpenNI2\Drivers).

You can find the precompiled examples in the folder <path-to-your-openni2-installation-folder\Samples\Bin:

ClosestPointViewer.exe: creates an OpenGL window displaying the depth stream and showing the closest point to the camera
EventBasedRead.exe: opens a console and displays the results of the read events
MultiDepthViewer.exe: creates an OpenGL window displaying the depth stream coming from multiple cameras
MultipleStreamRead.exe: opens a console and displays the results of the read events from multiple cameras
MWClosestPointApp.exe: opens a console and displays the distance of the closest point to the camera
SimpleRead.exe: opens a console and displays the distance of the central point of the depth image
SimpleViewer.exe: creates an OpenGL window displaying the color/depth combined stream (key 1), the depth stream (key 2) and the color stream (key 3)
NiViewer.exe: advance test GUI [available in the folder Tools]

Linux examples #

The driver comes with five examples demonstrating how to use it to retrieve and show color images, depth maps and point clouds.

SimpleMultiViewer: creates an OpenGL window displaying the color/depth combined stream (key 1), the depth stream (key 2) and the color stream (key 3)
SimpleColor: [requires OpenCV] creates an OpenCV window displaying the color stream
SimpleDepth: [requires OpenCV] creates an OpenCV window displaying the depth stream
SimpleRegistered: [requires OpenCV] creates two OpenCV windows displaying the synchronized color and depth streams
SimplePointCloud: [requires OpenCV] creates an OpenCV 3D window displaying the color stream mapped on a 3D point cloud

It is possible to execute the precompiled OpenNI2 examples (see Windows) manually copying the driver file libzeddriver.so in the folder <path-to-your-openni2-installation-folder\Samples\Bin\OpenNI2\Drivers> and <path-to-your-openni2-installation-folder\Tools\OpenNI2\Drivers>

NiTE 2 integration #

The ZED OpenNI2 driver is compatible with the NiTE v2.2 framework. To execute the precompiled example available with NiTE simply copy the ZED driver file (libzeddriver.so [Linux] or libzeddriver.dll [Windows])in the folder Samples/Bin/OpenNI2/Drivers under the root of the NiTE installation folder. NiTE will automatically load the new driver searching for a connected ZED camera when starting.