Live Link for Unity sample

The ZED Live Link for Unity sample is an interface between the SDK Fusion module and Unity. It sends fused skeleton data to the engine so that 3D avatars can be animated in Unity using one or several ZED cameras working together.

The data is broadcast via UDP on a specified port, and can be received using the assets of the paired Unity project.

Compatibility #

Dependencies #

  • ZED SDK v4.x, available on stereolabs.com
  • Unity 2021.3 and more recent versions, available on unity.com
    • It will probably work well with older versions, as its core is a UDP receiver that animates a humanoid avatar but has not been tested with them.

Difference with the ZED Unity Plugin #

The ZED Unity plugin, available here, is a full integration of the SDK features in Unity, except for the multi-camera data fusion. It comes as a Unity Package, allowing dialogue between your project and all the ZED SDK capabilities.

This sample comes as an alternative and a complement, allowing your Unity project to receive only Body Tracking data, from one or several cameras. The Unity project provided does not implement any dialogue with the SDK, only being able to receive data from the sender tools (not the SDK directly).

Content #

Sender tools #

The ZED Unity Live Link and ZED Unity Live Link Fusion projects are the C++ samples responsible for sending the data from the SDK to Unity via UDP.

  • The ZED Unity Live Link Monocam sample directly calls the SDK and sends only the Body Tracking data to Unity.

  • The ZED Unity Live Link Fusion sample uses the Fusion module of the SDK to combine the Body Tracking data from several cameras, via a direct USB connection or ZEDHub.

Unity project #

The Live Link for Unity project comes with some scripts to receive skeleton data, a sample scene to discover it and a 3D avatar to populate said scene.

Scenes :

  • Skeleton Fusion : main scene, simple space showcasing the necessary elements to receive skeleton data from the sender and animate some 3D avatars.
  • Upper Body Tracking : Implementation of a way to use only the upper part of the body (from the spine and above) to animate the avatar.

Main scripts :

  • ZEDStreamingClient : Manage the reception of data from the sender.
  • ZEDCommon : Defines the structure of the body data received and used in the sample.
  • ZEDSkeletonTrackingManager : Manages the display of the 3D avatars by processing the data received by the ZEDStreamingClient at each reception.
  • SkeletonHandler : Main script to manage the “data” part of each body detected.
  • ZEDSkeletonAnimator : Script to manage the animation of each avatar, using the SkeletonHandler data.

Main prefabs :

  • FusionManager : The main manager, with the ZEDStreamingClient and ZEDSkeletonTrackingManager components attached.
  • Unity_Avatar : Sample avatar using Unity’s Starter Assets 3D humanoid model.

Quick Start #

  • First, download and install the latest version of the ZED SDK on stereolabs.com (Minimum requirement: ZED SDK v4.0).
  • Download either the livelinkforunity.unitypackage or the Unity project from the Releases page along with the SenderTools.zip archive.
  • Unpack the archive at a location of your choice.
  • Generate either or both of the projects in SenderTools (fusion or mono) using CMake.
  • Open the main.cpp file and set the servAddress and servPort variables with your desired address and port.
    • The default values are 230.0.0.1 for the IP and 20001 for the port.
    // ----------------------------------
    // UDP to Unity----------------------
    // ----------------------------------
    std::string servAddress;
    unsigned short servPort;
    UDPSocket sock;

    sock.setMulticastTTL(1);

    servAddress = "230.0.0.1";
    servPort = 20001;

    std::cout << "Sending fused data at " << servAddress << ":" << servPort << std::endl;
    // ----------------------------------
    // UDP to Unity----------------------
    // ----------------------------------
  • Build the sample and execute it, passing your calibration file (see the documentation about ZED360 for more details) as argument, if you’re using the Fusion sender, or either nothing or your SVO file for the mono-camera sender.
> path/to/the/ZED_Sender_Fusion.exe path/to/the/calib_file.json

> path/to/the/ZED_Sender_Monocam.exe [path/to/the/file.svo]
  • Import the Unity package in your project or just open the project provided on this repo.
  • Check that the Port and Multicast IP Address variables of the ZED Streaming Client script on the Fusion Manager prefab are set to the same values as in the main.cpp set previously.

  • Run the scene and you should see avatars moving in Unity !

Main settings in Unity #

These are located on the ZED Skeleton Tracking Manager script in the Fusion Manager prefab.

  • Enable Avatar: controls the visibility of the 3D avatar.
  • Avatars : Array of 3D avatars randomly chosen when detecting a new person.
    • All of Unity Humanoid avatars should be compatible, provided they are made into a prefab with a ZEDSkeletonAnimator component attached.
  • Display SDK Skeleton : controls the visibility of the stickman view of the SDK keypoints. Whereas the 3D avatar is animated using local rotations derived from the keypoints, this shows the actual positions of the keypoints detected by the SDK.

For a complete description of the settings, please refer to the ZED Plugin for Unity’s Body Tracking section.