How to Use YOLO with the ZED

Introduction #

Download the YOLO samples at ZED YOLO Github

Other samples using OpenCV DNN or YOLOv5 with the TensorRT API in C++ or PyTorch can be found in the main ZED SDK repository.

These samples are designed to run state-of-the-art object detection models using the highly optimized TensorRT framework. Images are captured with the ZED SDK to detect 2D bounding boxes using YOLO, and the ZED SDK then extracts 3D information (localization, 3D bounding boxes) and performs tracking.

YOLO v5, v6, v7, v8, v9, v10, v11, v12 using TensorRT and C++ #

There are two main ways of running a YOLO ONNX model with the ZED and TensorRT:

  1. [Recommended] Use the OBJECT_DETECTION_MODEL::CUSTOM_YOLOLIKE_BOX_OBJECTS mode in the ZED SDK API to natively load a YOLO ONNX model. The inference code is fully optimized and internally uses TensorRT. The output is directly available as 2D and 3D tracked objects. This is the easiest and most optimized solution for supported models. ONNX to TensorRT engine generation (optimized model) is automatically handled by the ZED SDK.

    The model format is inferred from the output tensor size. If a future model uses a similar output format to the supported models, it should work.

  2. [Advanced - for unsupported models] Use external code for TensorRT model inference, then ingest the detected boxes into the ZED SDK. This approach is for advanced users, as it requires maintaining the inference code. It is suitable for models not supported by the previously mentioned method.

Exporting the model to ONNX (mandatory step) #

Refer to the documentation: https://www.stereolabs.com/docs/yolo/ to export your YOLO model to ONNX for compatibility. Depending on the version, there may be multiple export methods, and not all are compatible or optimized.

You can use the default model trained on the COCO dataset (80 classes) provided by the framework maintainers, or a custom-trained model.

In the folder cpp_tensorrt_yolo_onnx_native, you will find a sample that runs an ONNX model exported from a YOLO architecture using the ZED with the C++ API.

Python Version: Native Inference #

In the folder python_tensorrt_yolo_onnx_native, you will find a sample that runs an ONNX model exported from a YOLO architecture using the ZED with the Python API.

C++ Version: External Inference #

In the folder cpp_tensorrt_yolo_onnx, you will find a sample that runs an ONNX model exported from a YOLO architecture using the ZED with the C++ API, leveraging external TensorRT inference code and a bounding box ingestion function.

YOLOv8 PyTorch #

In the folder pytorch_yolov8, you will find a sample that interfaces the PyTorch Ultralytics package with the ZED SDK in Python.

This sample demonstrates how to detect custom objects using the official PyTorch implementation of YOLOv8 from a ZED camera and ingest them into the ZED SDK to extract 3D information and track each object.

Features #

  • The camera point cloud is displayed in a 3D OpenGL view
  • 3D bounding boxes around detected objects are drawn
  • Objects classes and confidences can be changed

Training your own model #

This sample can use any model trained with YOLOv8, including custom trained one. For a getting started on how to trained a model on a custom dataset with YOLOv5, see here https://docs.ultralytics.com/tutorials/train-custom-datasets/

Support #

If you need assistance go to our Community site at https://community.stereolabs.com/