Background Subtraction with UE5

In this tutorial, you will learn how to perform background subtraction with a ZED Camera and the NEURAL Depth Mode.

What is Background Subtraction? #

Background subtraction is a way of eliminating the background from image. To achieve this we are using the Depth map generated by the SDK to remove the background of an image. Everything further than a certain distance from the camera will be removed.

General Settings #

Now we’ve got to add a Blueprint from the ZED plugin. But by default, Content from plugins is hidden. To fix this, click on View Options at the bottom right of the Content Browser and enable Show Plugin Content.

Now click on the folder icon beside Content and click on Stereolabs Content to switch to the plugin’s content folder.

In the Content Browser, go to Plugins -> Stereolab Content -> ZED -> Blueprints and drag a BP_ZED_Initializer into the scene. This is the object that sets up your camera and handles communication with your app.

  • Select the BP_ZED_Initializer blueprint, and in the ZED section, uncheck the Show Zed Image parameter. This parameter has to be disabled so we can see the 3D scene and not the zed’s image in fullscreen.

  • In the Init Parameters section, set Depth Mode to NEURAL. This mode is mandatory in order to have accurate background subtraction.

📌 Note: The NEURAL depth mode is AI-based, therefore the AI model has to be downloaded and optimized when using this mode for the first time. If it has not been optimized yet, the plugin will start the optimization itself. It can take some time so we recommend using the ZED Diagnostic tool to optimize it beforehand.

Remove the background #

We are using the depth map to eliminate the background. Everything further than a certain distance from the camera will be removed. That’s why using a very accurate depth mode is important.

In UE5, there are 2 ways of doing that :

  • Adding a virtual object closer to the camera than the background. That’s what is done for the Weather forecast demo. A plane with a weather map place on it is added in the scene, just behind the person.

  • Using the Depth clamp Threshold parameter. This parameter will discard all the pixels of the image with a depth higher than the threshold. It can be useful if the background is a 3D scene which is further than the background of the image.

This parameter can also be modified at runtime, on the ZEDCamera actor, generated at runtime under ZEDPawn, letting you adapt the threshold in real time.