Fusion

The Fusion API of the ZED SDK offers developers the ability to create applications using data from multiple cameras with ease. This module handles time synchronization and geometric calibration issues, along with 360° data fusion with noisy data coming from multiple cameras and sensor sources.

Overview #

The Fusion module extends the functionalities of the ZED camera, without changing the way developers use it. It provides fine control over each camera and enables simultaneous utilization of multi-camera data. The Fusion API is continually evolving to introduce new features and enable the development of next-generation applications.

Developers can easily integrate the Fusion API into their existing applications. The module operates based on the Publish/Subscribe pattern, where the connected ZED cameras publish data, and the Fusion API subscribes to it, making data transmission seamless.

Overall, the Fusion API is a powerful tool that gives developers unprecedented control and versatility over their camera data, making it easier to build innovative applications that leverage multiple camera inputs.

Workflow #

To support most setups, the Fusion API can operate with two different workflows:

	LOCAL	NETWORK
Presentation	Fully local application, where all ZED cameras are connected to the same host computer which runs the publishers and the subscriber.	Network (edge computing) oriented application based on ZED Boxes, where each publisher and the subscriber runs on different host machines.
Diagram
Pros	- Low latency - Efficient shared memory protocol (even for GPU memories) - Can achieve high FPS with high-end hardware - Simple setup for development	- High availability (no single point of failure) - No limitations on the number of publishers - Large variety of ZED boxes available to fit your computation requirements - Adapted for large scale scenarios - Remote monitoring of your system with ZED Hub
Cons	- Requires a powerful GPU (large GPU memory) to support the execution of multiple camera instances - Requires high USB bandwidth to connect all of your cameras - Can be limited by camera cable length	- Network data transfer introduces a small latency (< 5ms), depending on the network configuration

📌 Note: The Fusion module is lightweight (in computation resources requirements) compared to the requirements for camera publishers.

Available modules: #

Currently, the Fusion API integrates the following ZED SDK modules:

Object detection
Body tracking
Spatial mapping
Positional tracking with external GNSS (refer to this page for additional details on this fusion module)

Getting Started #

To achieve optimal performance and accurate results when using the Fusion API of the ZED SDK, it is crucial to calibrate your camera array. The Fusion API requires the knowledge of each camera’s position to create a common 3D world representation. Therefore, the calibration process needs to be precise; otherwise, the results obtained from the Fusion API might be inconsistent.

To simplify this process, we have developed a tool called ZED360. You can find more information on its usage in its documentation. By utilizing ZED360, you can accurately calibrate your camera array, ensuring that you get the best possible results from the Fusion API

There are various calibration methods that you can use depending on your setup, and as the Fusion API continues to evolve, more options will be added.

The calibration process is critical, as accurate calibration is necessary to obtain reliable and precise results from the Fusion API. Proper calibration enables you to leverage the full potential of the Fusion API and its versatility.

Try it out #

Once the calibration is done, you can try out the Fusion with our C++ and Python multi-camera samples, or in Unity and Unreal Engine via their Live Link implementations.

Configuration files #

To accommodate varying workflows and provide the ability to update without requiring changes to application code, a configuration file has been designed to contain all the necessary data for the Fusion API of the ZED SDK. The ZED SDK includes a support function that enables direct parsing of the configuration file and API setup based on its contents.

The configuration file contains the information for the publisher to open the camera and for the subscriber to connect to the publisher as well as the camera pose. Data are packed per camera, referred to by their serial number. The input part is separated into:

zed: the way the publisher can open the camera. The parameters are based on the API Input_Type, type is the API enum INPUT_TYPE and configuration is a string containing input data.
fusion: the way the subscriber can contact the publisher. If your workflow is local, simply set the type to INTRA_PROCESS, no need to define configuration. In case your workflow is over the network, set the type to LOCAL_NETWORK and add configuration|ip as a string and configuration|port as a number.

The file also contains the camera’s world data. It is defined by its rotation (radians) and its translation (meters). The position should be defined in UNIT::METER and COORDINATE_SYSTEM::IMAGE, the API read/write functions will handle the metric and coordinate system changes.

Configuration files are comprised of a list of parameters for each camera that should be configured in that calibration. You can find details on each attribute below:

Configuration File Example

{
    "12345678": { //← serial number of camera (you can find it in the ZED Explorer tool)
        "input": {
            "zed": {
                "type":  // could be "USB_SERIAL", "USB_ID", "GMSL_SERIAL", "GMSL_ID", "SVO_FILE" or "STREAM"
                "configuration": // for type X_ID: camera ID, for type STREAM: IP [and port] formatted as "IP:PORT", for type SVO_FILE, path to SVO
            },
            "fusion": {
                "type": // "LOCAL_NETWORK": (ZED Hub) Sender of this camera not running on the same machine than Fusion
                        // "INTRA_PROCESS": (Local workflow or reading SVOs) Sender of this camera & Fusion running on the same machine
                "configuration": { // needed only in case of 'LOCAL_NETWORK'
                    "ip": "192.168.1.234", // the ip of the edge device that runs the camera
                    "port": 30000 // the port the data is streamed on
                }
            }
        },
        "world": {
            "rotation": [ // orientation of the camera in radians
                0,
                0,
                0
            ],
            "translation": [ // position of the camera in meters
                0,
                0,
                0
            ]
        }
    },
// ...
// [same pattern for the other cameras]
// ...
}