How to Use TensorFlow 1 with ZED

Introduction

The ZED SDK can be interfaced with TensorFlow for adding 3D localization of custom objects detected with Tensorflow Object Detection API.

In this tutorial, we will show you how to detect, classify and locate objects in 3D using the ZED stereo camera and TensorFlow SSD MobileNet inference model.

Installation

The 3D Object Detection project depends on the following libraries:

  • Python 3
  • CUDA
  • ZED SDK
  • ZED Python API
  • cuDNN
  • Tensorflow v1
  • Tensorflow Object Detection API
  • OpenCV

ZED SDK

If you want to use a virtual environment for your developments, create one and make sure to activate it

# This will create a ~/tensorflow directory containing all the packages you wish to install in your environment
virtualenv ~/tensorflow
# Activate your virtual environment 
source ~/tensorflow/bin/activate

Install the ZED SDK and ZED Python API.

cuDNN

Install cuDNN. Check the support matrix for the corresponding CUDA and driver version.

Tensorflow Object Detection API

Install Tensorflow with GPU support by reading the following instructions for your target platform.

# GPU package for CUDA-enabled GPU cards
python -m pip install --upgrade tensorflow-gpu==1.15

Install Tensorflow Object Detection API by following these instructions and download the model repository.

git clone https://github.com/tensorflow/models

Test that you have correctly installed the Tensorflow Object Detection API by running the following command:

python object_detection/builders/model_builder_test.py

Note: If you get an import error, make sure that tensorflow/models/research/slim directories have been added to PYTHONPATH. This can be done by running the following command:

# From tensorflow/models/
export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim

Running 3D Object Detection

Download the sample project code from GitHub. Make sure the virtualenv is active.

source ~/tensorflow/bin/activate

Run the code with python3. You should be detecting objects captured by your ZED camera using MobileNet and localizing them in 3D.

python3 object_detection_zed.py

Testing other models

In this example, we’re using the computationally efficient MobileNet model for detecting objects. You can change this by updating the MODEL_NAME variable and selecting another one from Tensorflow model zoo. These models will be downloaded and extracted automatically. For example, a ResNet model can used by changing MODEL_NAME to :

# Full model name required
MODEL_NAME = ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03

Testing a custom model

Other custom object detection models can be loaded by modifying the PATH_TO_FROZEN_GRAPH, variable typically called frozen_inference_graph.pb.

Notes

CUDA / cuDNN version

Please refer to the Tensorflow compatibility table to know the appropriate cuDNN and CUDA versions for a given Tensorflow version. At the moment of writing, Tensorflow requires CUDA 9.0 and cuDNN 7.

Architecture

This sample uses 2 threads, one for the ZED images capture and one for the Tensorflow detection. While it may seem complex at first, it actually solves 2 issues:

  1. Performance is increased, as depth computation is done in parallel to inference.

  2. Tensorflow and the ZED SDK uses CUDA GPU computation and therefore requires the use of CUDA contexts. Since we currently can’t share the CUDA Context between the ZED and TF, we have to separate the GPU computation. Each CUDA context must therefore have its own thread.