Body Tracking with UE5

BP_ZED_Initializer

Put the BP_ZED_Initializer in your scene to configure all the features of the ZED camera. This actor is a direct reference to the C++ code of ZEDInitializer.cpp. Pay special attention to some parameters :

• Input Type : will the SDK compute data from a ZED camera, a SVO file, or a stream input.
• Resolution, FPS, Depth Mode : Main parameters of the ZED camera. See the API Reference for more details. Default values HD 1080p, 30 fps and Depth Mode Ultra will work the best in most cases.
• Loop and Real Time : If you’re using a SVO file, you will want to tick these two to true in most cases. It enables SVO looping and syncing the SDK processing time with the SVO timestamps.
• Depth Occlusion : If enabled, the virtual objects will be hidden by the render of the real scene, using the depth of each pixel of the camera.
• Show Zed Image : If enabled, the ZED Camera/SVO video stream will be rendered in fullscreen in the level viewport at runtime. You may want to leave it on in some cases (MR applications mostly) but in this tutorial, we want the virtual scene to occupy the screen.

BP_BodyTracking_Visualizer

This blueprint manages the whole avataring process, from enabling object detection to sending the skeleton data to the animation blueprint.

• Enabling the Object Detection :

• Automatically : If the “Start Object Detection Automatically” option is ticked, the object detection will start as soon as the ZED camera is ready.

• Manually : If it is not ticked, it can be started in the details panel manually.

• Watch for the retrieve object callback

• Put a small viewport with the image from the ZED Camera / SVO file in the main viewport.

• Loop on each retrieved object and check its tracking state.

• If it is a new detection, spawn an avatar.

• If it is an already tracked skeleton, find the avatar in question in the avatars actor map.

• Set Animation Blueprint’s Object Data and Skeletal Mesh variables (see AnimNode_ZEDPose.cpp below.)

BP_ZED_Manny & ABP_ZED_Manny

The BP_ZED_Manny Actor Blueprint includes a skeletal mesh using a specific animation blueprint, the ABP_ZED_Manny. The procedure to create your own is detailed below in this article.

NB : As mentioned in this guide, for now, the skeleton used must be in T-Pose to have as correct as possible correspondence of the movements.

The ABP_ZED_Manny is fed a SlObjectData structure containing all the data about the skeleton, mainly key points positions, bones orientations, and location of the skeleton. A Control Rig node might be activated afterward, sticking the feet to the ground when nearing it.

ZED Pose - Animation Blueprint node

This node is generated by the AnimNode_ZEDPose.cpp file, do not hesitate to take a peek at it. This is the core of the avataring, it applies the rotations of the bones of the SDK skeleton to the Unreal avatar bones (using the remap asset).

• Object Data : SlObjectData structure used to retrieve the location and orientations of key points and bones from the SDK. It must be set externally (e.g. from the BP_BodyTracking_Visualizer). Explore StereolabsBaseTypes.h for more details on this structure.
• Remap Asset : This map of names is used to make the correspondence between the bones' names in the SDK and the ones of the avatar.
• Mirror on Z Axis : Enable mirror-mode for the avatar. If you raise your right hand, it will raise its left.
• Height Offset : Can be used to apply an external offset to the character Z location (height).
• Rotation Slerp Intensity : Intensity of the smoothing applied to the avatar movements. You can adjust it to adjust latency on movements and jittering.
• Root Location Lerp Intensity : Intensity of the smoothing on the location of the root of the skeleton. Used to avoid sudden displacement which can happen in case of unstable data from the camera.
• Skeletal Mesh : Used to give the skeletal mesh to the C++ script, which then manipulates its Pose.

Importing and using your own 3D avatar and rig as base [Deprecated]

One 3D Avatar is already available in the Example project but you can also import your own model. First, your 3D model needs to be rigged, otherwise it won’t be possible to animate it.

For now, our plugin allows to animate correctly an avatar if it is imported in the right format. We need the avatar to respect two conditions:

• The first condition is on the avatar’s reference pose. The reference pose is the default pose of the avatar you can see when you open the skeleton asset window. We need this reference pose to be in T-Pose, which corresponds to the pose where the avatar is standing up with its arms held horizontally.

• The second condition is on the avatar’s global orientation when imported. We want to start from a known orientation, so we need the avatar to be facing the +X direction in its reference pose.

To respect the first condition, you might need to change your avatar’s reference pose if it isn’t already a T-Pose. You can change its reference pose using Blender, for instance.

Moreover, in order to meet the second condition, you’ll have to tick the checkbox Force Front X Axis to force the avatar to look at the +X direction. This way we make sure the avatar starts with a known orientation.

Importing procedure with Blender

1. Import the mesh into Blender.
2. Make sure the mesh is oriented following Blender defaults parameters: Facing -Y, Z up.
3. Rig the mesh.

• If you’re using Mixamo to make your skeleton, ensure your scale is applied before exporting your fbx/obj to Mixamo.

4. Pose the mesh in T-Pose.
5. When exporting from Blender, ensure your scale / transforms are applied beforehand, and use the default FBX export settings.

• You can leave “Add leaf bones” checked or not, depending on your needs.

6. Import to Unreal, checking the following settings :

• Skeletal Mesh
• Convert Scene
• Force Front XAxis
• If you’re using a T-Pose which is not the rest pose of your rig (e.g. from Mixamo):
  • Update Skeleton Reference Pose
  • Use T0 As Ref Pose
    

Creating an Animation Blueprint

Once you imported the avatar, the next thing to do is to create an Animation Blueprint. For that, Right click on the Skeletal Mesh you just imported, and select Create -> Anim Blueprint.

Now, you need to configure this Anim Blueprint to receive the skeleton data from the ZED SDK and to apply it correctly onto the avatar.

• Open the Anim blueprint and add a ZED Pose component.
• Create a variable for each input of this component and link it to the Output Pose node.

Setting up the Remap Asset

Your avatar joints probably don’t have the same name than the one given by our plugin. The Remap Asset variable is used to match the joints from the ZED SDK with the joints of your avatar, exactly as the Livelink Remap asset.

• Select the Remap Asset variable and, for each of the 34 joints from the ZED SDK, add its matching name from your avatar.

For example, “LEFT_SHOULDER” is mapped to “LeftForeArm”, or “LEFT_KNEE” to “LeftLeg”.

Creating an Actor blueprint

Now, we have all the pieces to create a Blueprint than can be instantiated in the scene and animated using the ZED SDK skeleton data.

• Create an Actor Blueprint and add a Skeletal Mesh component.
• Set the Skeletal Mesh and the Anim class fields with the skeletal mesh you just imported and the Anim Blueprint previously created.

Your Actor is now ready to be automatically instantiated in the scene.

Setting up the Body Tracking Manager

Now, you need to configure the Body Tracking Manager so it will create a new instance of your Actor blueprint for each new detection in the scene and share the keypoint data with its anim blueprint.

• Click on Edit BP_BodyTracking_Visualizer in the world hierarchy
• In the Event Graph, modify the parts of the blueprint highlighted in the following screenshot with your own Instance of AnimBlueprint.
• It will share the skeleton data from the ZED SDK to the ZED Pose node of the Anim blueprint for preprocessing.
• Then, the preprocessed data will be used to animated your Avatar in real-time.

1. Import the mesh into Blender.
2. Make sure the mesh is oriented following Blender defaults parameters: Facing -Y, Z up.
3. Rig the mesh.

• If you’re using Mixamo to make your skeleton, ensure your scale is applied before exporting your fbx/obj to Mixamo.

4. Pose the mesh in T-Pose. (optional)
5. When exporting from Blender, ensure your scale / transforms are applied beforehand, and use the default FBX export settings.

• You can leave “Add leaf bones” checked or not, depending on your needs.

6. Import to Unreal, checking the following settings :

• Skeletal Mesh
• Convert Scene (optional)
• Force Front XAxis (optional)
• If you’re using a T-Pose which is not the rest pose of your rig (e.g. from Mixamo): (optional)
  • Update Skeleton Reference Pose
  • Use T0 As Ref Pose