Getting Started with Unreal Engine and ZED

Introduction

The ZED Plugin for Unreal allows developers to build AR/MR applications within Unreal Engine 5. It provides access to stereo video passthrough, depth sensing, positional tracking, spatial mapping and object detection features from the ZED SDK.

Compatibility

ZED Cameras

The Object Detection and Body Tracking features will only work with ZED2 and more recent cameras.

Version 1.0

This version of the plugin is compatible with UE 5.0.3 and UE 5.1. For now, it only interfaces with DirectX 11, making the compatibility outside of Windows platforms limited. Support of Vulkan and DirectX 12 will be added in a further update.

Version 1.1

This version of the plugin is compatible with UE UE 5.1. Support of DirectX 12 has been added, along Body Tracking improvements.

How it works

The plugin uses our C language wrapper to interface with the SDK. This allows the usage of all the SDK features mentioned above, with either a video feed from a ZED Camera (via USB or streaming) or by reading a SVO file.

Main differences with UE 4.21 ZED plugin

This version of the plugin does not require a custom version of the Unreal Engine.

However, some features are not yet available :

  • Virtual Reality headsets compatibility
  • Lights interactions between virtual lights and the real world.

Available samples

  • L_BackgroundSubstraction : Background subtraction with plane occlusion or depth clamping.
  • L_BodyTracking : Body tracking in a virtual scene, with the Unreal Mannequin as an avatar.
  • L_CameraTracking : Camera tracking demo in a virtual scene.
  • L_Demo_Stage : Body tracking with legs inverse kinematics to improve the foot/ground interaction.
  • L_DynamicCrosshairOnMouse : Depth dynamic crosshair hovering on the environment, without spatial mapping.
  • L_NewHitTest : Depth-based hit test with ball stopping on the environment without spatial mapping.
  • L_ObjectDetection_2D : Object detection (human body model) with 2D frames.
  • L_ObjectDetection_3D : Object detection (human body model) with 3D boxes.
  • L_OpenNewLevel : Procedure to change active level, ZED-wise.
  • L_Passthrough : Simple AR passtrough vision showing a virtual object in a real scene, with depth occlusion.
  • L_PlaneDetection : Showcase of the SDK plane detection feature.
  • L_PointCloud : Point cloud mapping of the environment.
  • L_SkeletonAvatar : Visualization of the skeleton tracking data from the SDK.
  • L_SkeletonTrackinSpatialMapping : Usage of both the skeleton tracking and the spatial mapping features.
  • L_SpatialMapping : Visualization of the spatial mapping of the environment, and potentially creation of a static mesh out of the mapping.