Area Memory and Relocalization

Area Memory refers to human memory for spatial information, such as the geographical layout of a town or the interior of a house. As we navigate the world, we store information about our surroundings that form a coherent spatial representation of the environment in memory. Similarly, the ZED builds and updates a spatial representation of its surroundings when discovering an environment.

This representation is stored in the form of a heuristic map called an Area file. Area files are compact and efficient representations of an environment that are not meant to be visualized. Area Memory is useful to:

  • Localize multiple devices in the same space
  • Improve positional tracking by correcting drift when revisiting a known area
  • Ensure a consistent experience in a known area

Area Memory for Drift Correction #

Area Memory is enabled by default in PositionalTrackingParameters. While moving the ZED camera in its environment, key images and contextual information will be saved. When the tracking detects an already-visited zone, it will compute an updated position estimation that cancels eventual drifts. This solution is known as a loop-closure.

The loop-closure compensation behavior can generate a sudden jump in the returned positions. To smooth this jump, PositionalTrackingParameters.pose_smoothing can be used to spread the drift correction across the upcoming estimations.

Memorizing an Area #

During tracking, the ZED will incrementally build an internal representation describing the viewed area. To save this representation as an Area file, use saveAreaMap() during tracking or end the tracking session with disablePositionalTracking() and add a path as an argument.

// Disable positional tracking and save an Area file
zed.disablePositionalTracking("filename.area");

# Disable positional tracking and save an Area file
zed.disable_positional_tracking("filename.area")

// Disable positional tracking and save an Area file
zed.DisablePositionalTracking("filename.area");

The Area file contains the position of the stationary World Frame created at the beginning of the session. Saving and loading an area file allows to maintain and share a fixed reference frame between different sessions.

Loading an Area #

To load an area file, simply add the path of the file to PositionalTrackingParameters and enable tracking.

PositionalTrackingParameters tracking_params;
tracking_params.area_file_path = "filename.area";

tracking_params = sl.PositionalTrackingParameters()
tracking_params.area_file_path = "filename.area"

PositionalTrackingParameters trackingParams = new PositionalTrackingParameters();
trackingParams.areaFilePath = "filename.area";

When tracking is enabled, the ZED will start searching for spatial similarities between the current and recorded area. Once it recognizes the space, POSITIONAL_TRACKING_STATE will switch from SEARCHING to OK and positional tracking will start using the previously recorded World Frame as a reference frame.

In certain cases, the camera may not be able to recognize its surroundings. You will need to create a new Area file if:

  • The spatial layout of the scene has changed or significant objects have moved.
  • Area has been covered very quickly or with specific camera angles.
  • Area has many blank walls or identical surfaces.
  • Starting position is far from the trajectory used during spatial memorization.

Use Cases #

Area memory has three common use cases:

Improve positional tracking #

As with IMUs, motion estimation works well locally but small estimation errors accumulate slowly over time. Area Memory mode enables the ZED to learn and recognize its surroundings during use. When the device recognizes an area, it corrects its position in space and eliminates the positional drift that may have accumulated.

Drift correction can sometimes cause jumps in the absolute pose. The relative pose is not corrected. If your application requires a smooth pose, use getPosition() with Camera Frame as a reference frame. If you want to distribute the correction over multiple frames, you can create a correction buffer by calculating the difference between the corrected camera path (obtained using getPosition() in the World Frame) and the relative pose.

Ensure a consistent experience #

Some applications need to map an area first before running multiple times in this specific area. This is the case with AR applications or robots that navigate a home or a warehouse. In this case, using Area Memory enables the application to store a spatial representation of the environment for future use.

During new runs, the device can load the Area file previously recorded and recover the World Frame position and 3D data attached to it. Every virtual object or navigation map will appear in the same physical location, enabling a consistent experience between runs.

Localize multiple devices in space #

When multiple cameras are successfully relocalized against the same map, the devices can then share a common reference frame and real-world positions during a session.

The devices will localize themselves in the World Frame recorded in the Area file. A user can specify the position of this reference frame using an Initial World Transform, otherwise the World Frame will be located at the initial position of the camera when tracking with spatial memorization was started. Without area memory, different devices will not be able to understand their absolute position in the world and will track their position with respect to their own starting point.