Animating New Avatars

The Unreal project we provided you already contains avatars you can animate. This section will explain how to import a new avatar to make it work with our ZED Live Link plugin. You can find various avatars on Mixamo website for instance.

Before explaining in detail how to animate a new avatar, it’s important to understand the format of the skeleton data sent by ZED Live Link.

Live Link works with the notion of Subject, which is an individual stream of data, typically describing a given individual character. For a given Live Link subject, at a given frame, we output a total of 34 joints, in the following order:

Joint index Joint name
0 PELVIS
1 SPINE_NAVAL
2 SPINE_CHEST
3 NECK
4 CLAVICLE_LEFT
5 SHOULDER_LEFT
6 ELBOW_LEFT
7 WRIST_LEFT
8 HAND_LEFT
9 HANDTIP_LEFT
10 THUMB_LEFT
11 CLAVICLE_RIGHT
12 SHOULDER_RIGHT
13 ELBOW_RIGHT
14 WRIST_RIGHT
15 HAND_RIGHT
16 HANDTIP_RIGHT
17 THUMB_RIGHT
18 HIP_LEFT
19 KNEE_LEFT
20 ANKLE_LEFT
21 FOOT_LEFT
22 HIP_RIGHT
23 KNEE_RIGHT
24 ANKLE_RIGHT
25 FOOT_RIGHT
26 HEAD
27 NOSE
28 EYE_LEFT
29 EAR_LEFT
30 EYE_RIGHT
31 EAR_RIGHT
32 LEFT_HEEL
33 RIGHT_HEEL

Our Live Link plugin will feed local rotation and translation data for each of these joints. Local rotation for a joint is defined as its rotation relative to its parent joint. You can have an idea of the joint hierarchy on the previous image (a skeleton is basically a tree). We see that the root of this skeleton is the PELVIS, which has 3 children joints (SPINE_NAVAL, HIP_LEFT and HIP_RIGHT), etc. In addition to these local rotations (given in the form of a quaternion (x,y,z,w)) and translations, our Live Link plugin will give the root global rotation and translation.

All these joints data are expressed in Unreal coordinate frame, which is LEFT_HANDED_Z_UP. Each joint orientation will be expressed in the same coordinate frame you can see here for the avatar’s hips:

To animate a new avatar in the provided Unreal project, follow these steps:

1. Importing a New Avatar

For now, our plugin allows to animate correctly an avatar if it is imported in the right format. We need the avatar to respect two conditions:

  • The first condition is on the avatar’s reference pose. The reference pose is the default pose of the avatar you can see when you open the skeleton asset window. We need this reference pose to be in T-Pose, which corresponds to the pose where the avatar is standing up with its arms held horizontally.

  • The second condition is on the avatar’s global orientation when imported. We want to start from a known orientation, so we need the avatar to be facing the +X direction in its reference pose.

To respect the first condition, you might need to change your avatar’s reference pose if it isn’t already a T-Pose. You can change its reference pose using Blender, for instance. Let’s take the example of this robot avatar, that initially has an “A-Pose”:

You can change its reference pose this way:

  1. Select the Armature of your avatar in the Scene Collection and select the Pose Mode this way:

  1. For each bone that needs to be rotated, click on it and go to Pose/Transform/Rotate:

Once you did all the necessary rotations and that your avatar appears in T-Pose, you can define it as the new reference pose (which is called Rest Pose in Blender). Go to Pose/Apply and select Apply Pose as Rest Pose:

You can then export your avatar as an FBX object and its reference pose will be the T-Pose you defined.

When you import your avatar in Unreal, you’ll have to tick the checkboxes Update Skeleton Reference Pose and Use T0 As Ref Pose to make sure the avatar’s reference pose in Unreal is the one you just defined in Blender.

Moreover, in order to meet the second condition, you’ll have to tick the checkbox Force Front X Axis to force the avatar to look at the +X direction. This way we make sure the avatar starts with a known orientation.

Make sure, by checking the avatar skeleton asset, that its reference pose is in T-Pose, and is looking at the +X direction. This should look like this:

2. Creating a Remap Asset

Once you imported the avatar, the next thing to do is to create a Remap Asset. Indeed, your avatar joints probably don’t have the same name than the one given by our plugin. It’s for example the case with Mixamo avatars we used in the Unreal project you’re provided. Live Link gives the possibility to remap these bone names. Create a new Blueprint class, which parent class must be LiveLinkOrientationsRemapAsset.

Open this blueprint class and override the function “GetRemappedBoneNames”. There you must do a similar remapping than on the picture below, where bones explained in the first section must be mapped to their name in your imported avatar. The bone names in the “Switch on Name” node correspond to the bone names given by our Live Link plugin, and the bone names in the “Return Node” nodes correspond to your imported avatar bone names. The Unreal project we provided you has an example of such a Remap Asset useable for Mixamo avatars, called MixamoRemap.

3. Creating an Animation Blueprint

Data from Live Link can be applied directly inside Animation Blueprints.

When creating an Animation Blueprint, a dialog box will allow you to select its “Target Skeleton”. Choose the skeleton created when you imported your avatar.

This is achieved by creating a Live Link Pose node inside the AnimGraph of an Animation Blueprint:

  • Create a variable Live Link Subject and a variable Skeletal Mesh and put it in the AnimGraph. It refers to the name of the subject in Live Link to stream data from. In case several persons are detected, this variable will feed the right skeleton data to the right avatar. The Skeletal Mesh variable will be used to apply an offset to the avatar in order to make a stick on the ground as much as possible.
  • Link it to a Live Link Pose node, which will get the actual skeleton data from our Live Link Subject.
  • Link this node to the Output Pose, this way the Live Link skeleton data will directly animate our avatar.

In the “Details” menu of the Live Link Pose node, you have the possibility to select a retarget asset. This is where you must select the Remap Asset you created in the previous step, in order to remap correctly the Live Link skeleton data to your avatar bone names.

The AnimGraph should look like this:

You can refer to the existing Animation Blueprints in the Unreal project to configure yours the same way.

Next, you must select the “Avatar_livelink” asset already provided in the Unreal Project. The role of this asset is to define which avatar will be animated in the Unreal scene. In the “Skeletal Mesh” details menu, you must select the Skeletal Mesh you imported and want to use, and the Anim Class linked to this mesh that you previously created.

The Live Link Skeletal Animation component has be added to this Actor in order to drive its parameters with Live Link from a connected external source.

5. Configuring project level blueprint

Once the “Avatar_livelink” is set, last step is to open the Level Blueprint of the project. This Level Blueprint automatically connects the project to the Live Link Source at Play, and a Avatar_livelink Actor is instantiated for each skeleton sent by the Live Link plugin, in order to be rendered on the Unreal scene.

At the right side of the EventGraph, you will notice two nodes “Cast To Anim_livelink” and “SET Live Link Subject”.

The idea is to replace these nodes with a cast to your own Animation Blueprint you created for your avatar. Remove the two current nodes (or suppress their links to rest of the graph), and create a cast to your own Animation Blueprint, as well as a set to its “Live Link Subject” variable. Reproduce the same connections set before with your Animation Blueprint.

Once done, you’re ready to press play and see your avatar animated with Live Link data.