Getting Started with Unity and ZED

The ZED SDK Plugin for Unity allows developers to build interactive experiences with a first or third-person view. It provides access to body tracking, stereo video passthrough, depth sensing, positional tracking, and spatial mapping features within Unity. Examples of applications the ZED plugin makes possible are:

  • Building Mixed-Reality (AR/MR) interactive experiences where real and virtual objects blend.
  • Displaying Passthrough AR experiences in an Oculus Rift or HTC Vive.
  • Capturing VR applications from a third-person point of view in front of a Green Screen.

ZED Plugin for Unity #

The ZED Plugin for Unity is available as a Unity package through the Package Manager. It includes assets, scripts, and sample scenes to assist with development. Many of its core features, such as mixed reality, take minutes to implement thanks to its prefabs.

📌 Note: Before the release 4.1.1, the plugin was available as a .unitypackage file to install in your project. These versions, made for the ZED SDK up until 4.1, are available on the Releases page.

The package contains the following subdirectories:

  • XR: Configuration assets for XR management.
  • Editor: Contains scripts to add functionality to the Unity editor interface.
  • Prefabs: Contains the main prefabs to add a ZED camera to a scene: ZED_Rig_Mono and ZED_Rig_Stereo.
  • SDK: Contains files and resources to interface the ZED SDK with Unity.

📌 Note: The samples can be installed individually from the Package Manager.

Plugin Requirements #

   
ZED CameraGPUAR/VR
ZED, ZED Mini, ZED2, ZED2i
CUDA 10.2 capability or higher
GTX 1060 or higher is recommended
OpenVR, Oculus VR or OpenXR
OS
Unity Version
ZED SDK Version
Windows 10/11
Ubuntu 18-22
Unity 2021.3 and newer
ZED SDK 4.1
Previous versions are supported by previous plugins

How to Get Started #

To develop applications in Unity with your ZED, you’ll need the following things:

  • Download and install the latest ZED SDK.
  • Install the ZED SDK plugin through Unity’s package manager:
    • Navigate to Window -> Package Manager in the Editor
    • Click the “+” button at the top left and select “Add package from git URL…”
    • Copy this URL into the field and hit Add:
https://github.com/stereolabs/zed-unity.git?path=/ZEDCamera/Assets
  • Wait for the package to install (it can take some time, depending on your connection)
  • Install the samples you want to try out/use through the Package Manager
    • Select the “ZED SDK” package
    • Go to the “Samples” tab and import the samples you want.
    • Consider importing the Assembly Definition, it potentially improves the compile time and editor responsiveness at no cost.
  • Read the Basic Concepts and Build Your First AR/MR App guides to get started.
  • Explore the Samples included in the plugin.
  • The Main Scripts part of the documentation introduces the important scripts of the plugin and their parameters.

Step 1: Installation #

  • Install the ZED SDK and restart your computer.

Step 2-1: Using the Universal Render Pipeline or the High-Definition Render Pipeline #

To use URP or HDRP, you must either create a new Project that uses one of them or upgrade an existing one.

Create a new URP/HDRP Project #

  • Open Unity Hub and Click the New Button.
  • In the Template section, select the Universal 3D or High Definition 3D template.
  • Click Create. Unity automatically installs the necessary packages and sets up your project to use them.
  • Import the ZED plugin.

Upgrade to URP #

URP upgrade instructions
  • Go to Window -> Package Manager. Find and install Universal RP.

Now you need to configure the URP. To do this, you have to create a Scriptable Render Pipeline Asset.

  • Right-click in the project window and select Create -> Rendering -> Universal Render Pipeline -> Pipeline Asset. You can also go to Assets -> Create -> Rendering -> Universal Render Pipeline -> Pipeline Asset.

Add this Render Pipeline Asset to your Graphics settings.

  • Go to Edit -> Project Settings -> Graphics. In the Scriptable Render Pipeline Settings, add the Universal Render Pipeline Asset you just created.

Upgrade all the materials to URP:

  • Go to Edit -> Render Pipeline -> Universal Render Pipeline and select either Upgrade Project Materials To URP Materials or Upgrade Selected Materials to URP Materials. These changes cannot be undone. Back up your project before you upgrade it.

For more details on how to set up URP, see the Universal Render Pipeline Documentation.

Upgrade to HDRP #

HDRP upgrade instructions
  • Go to Window -> Package Manager. Find and install High Definition RP.
  • To set up your project, use the HDRP Wizard available at Window -> Rendering -> HDRP Wizard.
  • In the HDRP section, click the Fix All button. HDRP is now set up in your project.
  • You might also need to click on Convert All build-in materials to HDRP to convert the materials to HDRP in your project.

For more details on how to set up HDRP, see the High Definition Render Pipeline Documentation.

Step 3: Import the Plugin #

  • Open your project.
  • Go to Window -> Package Manager. Click the "+" button at the top left and select “Add package from git URL…”.
  • Copy this URL into the field and hit Add:
https://github.com/stereolabs/zed-unity.git?path=/ZEDCamera/Assets
  • Wait for the package to install (it can take some time, depending on your connection)
  • Install the samples you want to try out/use through the Package Manager
    • Select the “ZED SDK” package
    • Go to the “Samples” tab and import the samples you want.
    • Also, import the Assembly Definition which potentially improves the compile time and editor responsiveness at no cost.

📌 Note: If you’re using the URP or HDRP render pipelines, you also need to Upgrade the materials (even if you created an URP/HDRP project before importing the plugin).

  • URP: Go to Window -> Rendering -> Render Pipeline Converter and convert all materials from Built-In to URP. This is not reversible, so proceed with caution.
  • HDRP: Go to Window -> Rendering -> HDRP Wizard and convert all materials. This is not reversible, so proceed with caution.

📌 Note 2: Some materials may appear as pink even after upgrading the materials of the project. They should work correctly at runtime, but in any case this can be fixed by assigning the corresponding “*_URP” material instead of the current one. To do so:

  • select the game object with the pink material
  • scroll down to its material in the inspector
  • click the three dots at the top right of the material component
  • click “Select Material”, this should take you to the material asset in the Projects view
  • Next to the material should be its URP counterpart
  • Assign the URP material by dragging it onto the gameobject.

Step 4: Try it out #

Navigate to Assets/Samples/ZED SDK and try out the available samples!