Getting Started with Unreal Engine and ZED

Introduction

The ZED Plugin for Unreal allows developers to build AR/MR applications with a first or third person view. It provides access to stereo video passthrough, depth sensing, positional tracking, and spatial mapping features within Unreal Engine.

Stereolabs Custom Engine for AR/MR

Building a mixed-reality experience with real-time spatial interactions is complex and technically challenging. To facilitate development of stunning MR applications with UE4, we’re sharing an optimized Unreal Engine for AR/MR along with the ZED Plugin for Unreal.

This custom modification of the Unreal Engine allows us to add several key technologies necessary to create believable and interactive AR/MR experiences:

• Improved Stereo Passthrough: We’ve added technologies such as Video Async Reprojection (VAR) to eliminate rotational latency and improve visual comfort.
• Interactive Collisions: Virtual elements such as lasers or balls can collide with both static and dynamic objects in the real-world without having to scan the environment first.
• Realistic Shadows and Lighting: Real-time depth and normal maps have been integrated in the rendering pipeline to offer a photorealistic integration of virtual content in the real-world. Virtual objects can cast shadows and project light on both static and dynamic real-world objects.
• Depth Occlusions: Occlusion of a virtual object by the real environment is required in order to trick human eyes into believing the virtual object is really present in the scene. By integrating long range depth sensing into UE4, virtual objects are naturally occluded by the surrounding environment.

Read how to Setup your environment and start using the ZED with UE4.