OptiTrack Unreal Engine 4 Plugin
The OptiTrack Unreal Engine Plugin allows you to stream real-time rigid body, skeleton, and HMD tracking data from Motive into Unreal Engine 4 (UE4). This article focuses on how to set up those different streaming workflows. For instructions on setting up the the motion capture system, please refer to the Getting Started guide instead.
The OptiTrack Unreal Engine Plugin is divided into two plugins:
- The OptiTrack - NatNet Streaming Client which allows you to stream rigid body and skeleton data from Motive onto Unreal Engine.
- The OptiTrack - OculusVR plugin which allows you to implement HMD tracking onto your scenes.
Once the plugins are enabled, they receive the streaming data from Motive through an OptiTrack Client Origin object in your Unreal Engine level.
This plugin is supported for the following versions of the Unreal Engine, and it also requires the corresponding version of Oculus runtime as indicated in the table.
|Plugin Version||UE Version||Oculus SDK|
As of OptiTrack Unreal Engine 4 Plugin 1.2.0, Oculus DK2, CV1, and Gear VR HMD models are supported
- 1 Motive Data Streaming Setup (Server)
- 2 Unreal Engine Setup (Client)
- 3 HMD Tracking Setup
- 4 Live Streaming
- 5 Wireless Multiplayer Setup
Motive Data Streaming Setup (Server)
First of all, follow the below instructions and set up the data streaming settings in Motive. Once this is configured, Motive will be broadcasting tracking data onto a designated network interface where client applications can receive them.
Streaming in Motive
Open the streaming settings in Motive and set the values below:
- Enable Broadcast Frame Data setting at the top
- From Local Interface drop-down menu, select a network address that you wish to stream the data over to.
- Select desired data types to stream under streaming options. Streaming of rigid body is required. Streaming of skeleton is optional for skeleton tracking. For tracking HMDs, disabling the Marker streaming is adviced.
- Set the Skeleton Coordinates to Local.
- When streaming skeletons, set the appropriate bone naming convention for the client application. (If the character uses the FBX naming convention, this will need to be set to FBX.)
- For streaming over a wifi network, setting the Transmission Type to Unicast is strongly adviced.
Unreal Engine Setup (Client)
Next step is to set up the client. Follow the below instructions to install and configure the Unreal Engine plugin to receive the streamed tracking data.
Setting Up the Plugin
- Download the plugin ZIP file.
- Extract the contents from the ZIP file.
- Open the OptiTrack folder, transfer the OptiTrackNatNet and OptiTrackOculusRift folders into the Unreal Engine plugin directory. By default, this directory is installed in the
C:\Program Files\Epic Games\4.##\Engine\Pluginsfolder. There will be other plugins installed in the folder already. Simply copy and paste the OptiTrack plugin folders into this directory.
- Open/Create a UE4 project.
- Under the Edit menu, open the Plugin Panel.
- Browse to Virtual Reality section and enable the OptiTrack plugins (OptiTrack - NatNet Streaming Client and OptiTrack - OculusVR) that are added to the folder.
Note: When using the Oculus Plugin, make sure to disable the built-in Oculus Rift (or the Gear VR) and Stream VR HMD plugins under virtual reality plugins category. Otherwise, the editor may crash, or they may end up having a higher priority than the OptiTrack Oculus Plugin, and if enabled, will be used in preference to it.
Client Origin Setup
OptiTrack Client Origin
The OptiTrack Client Origin enables the Unreal Engine (client) to communicate with the NatNet server where the Motive’s rigid body, skeleton, and HMD tracking data are streamed.
Add this object simply by dragging and dropping the OptiTrack Client Origin from the Modes panel into the scene. Once the OptiTrack Client Origin object is placed in the scene, its position and orientation will reconcile the global origin of Motive in Unreal Engine. In other words, the tracking data will be represented according to where this Client Origin object is positioned and how it is oriented.
Global Origin: Both position and orientation of the OptiTrackClientOrigin will represent the global origin of the tracking volume within Motive.
Connecting Unreal Engine to Motive
- [Motive] First of all, make sure the Broadcast Frame Data entry is checked in the Data Streaming pane of Motive.
- [UE4] Once the plugin is added and enabled in the project, OptiTrack Client Origin class will be available from the Modes panel.
- [UE4] Drag and drop the OptiTrack Client Origin into the scene.
- [UE4] Place the OptiTrack Client Origin to the desired tracking origin within the scene.
- [UE4] Select the instantiated OptiTrackClientOrigin object from the World Outliner panel.
- [UE4] In the Details panel, make sure its Auto Connect setting is checked. This configures the client origin to automatically search the network and connect to Motive.
- Now that the client origin is set, the client origin will attempt to connect to Motive and start receiving the tracking data whenever the scene is played.
Connecting to a designated IP address
Advance settings: Auto-initialize
Animating Rigid Body
OptiTrack Rigid Body Actor
- Actor objects in Unreal Engine can be animated using rigid body tracking data from Motive. Once the OptiTrack - NatNet Streaming Client plugin is enabled in the project, OptiTrack Rigid Body component will be available to use. By attaching this component to an actor, you can animate its child components according to the movement of a rigid body in Motive. Each rigid body component is given a Trackable ID value and rigid bodies with the matching Streaming ID will be used to translate and rotate the target actor in Unreal Engine. Please note that this component must be attached to the parent node to animate its child actors.
- You can also use the OptiTrack Rigid Body Actor, which will be available on the Modes panel. This actor is simply a blank actor with just the OptiTrack Rigid Body component attached to it, and you can use this to easily associate the objects within the scene:
- [UE4] From the Modes panel, search for OptiTrack Rigid Body Actor. Then drag and drop the actor into the scene.
- [UE4] Under this rigid body actor, attach the target actor that you wish to animate.
- [UE4] Set the relative location and rotation to zero on this target actor, which should be a child of the rigid body actor at this point.
- [Motive] In Motive, assign a value to Streaming ID property for the target rigid body.
- [UE4] In the properties of the OptiTrack Rigid Body Actor component, match the Tracking ID with the Streaming ID of the rigid body asset in Motive.
- Make sure both Motive and OptiTrack Client Origin is set up for streaming, hit Play, and the attached actor object will be animated according to the live-streamed rigid body tracking data.
Characters in Unreal Engine can be animated using skeleton tracking data in Motive. There are two different approaches to this: skeleton bone mapping and skeleton retargetting.
For plugin versions above 1.9.2, the skeleton component can be used to allow streaming of the skeleton via bone mapping. This means that the motion of each skeleton bone in Motive can be projected to the bones of the character model. Please note that this is different from skeleton retargeting, the bone segments in Motive get mapped directly to the character model (bone mapping) instead of being translated into something that is usable by a more abstract biped model (retargeting). The instructions on setting up the skeleton component are documented in the following page:
Skeleton tracking data can also be integrated into Unreal Engine via a 3D retargeting process. This can be accomplished through a third party software that retargets the 3D data to produce realistic motion in a scene. For more information, contact us.
Notes for IKinema LiveAction for UE 4.18.
Integrating HMD (Oculus DK2, CV1)
To integrate an HMD, make sure the OptiTrack – OculusVR is enabled in your project under the Plugins panel and follow the below steps to link the HMD tracking data.
Linking the HMD Tracking Data
- [UE4] Enable the OptiTrack – Oculus Rift plugin from the plugins panel.
- [UE4] Open the World Settings panel under the Window tab.
- [UE4] If not done already, create a new GameMode by adding a new blueprint under the Game Mode section.
- [UE4] Change the Default Pawn Class to OptiTrackHmdPawn. You may need to restart the editor before this change takes effect.
- [UE4] Select the OptitrackClientOrigin and access its settings under the Details panel.
- [UE4] Expand the OptiTrack properties section.
- [UE4] In the HMD Rigid Body ID entry, input the respective streaming ID of the HMD rigid body in Motive.
- [UE4] In the HMD Orientation section, make sure the axis orientation matches the axis orientation of the HMD rigid body in Motive.
- [UE4] Now the tracking data is linked to the Unreal Engine project, and the HMD will display the scene when it is played.
Integrating HMD (Android: GearVR)
The plugin also support integration of GearVR HMDs. The HMD tracking data can be streamed over a WiFi network, and in order to do so, a router with sufficient bandwidth must be used. The required bandwidth will vary depending on many factors (e.g. router type, the number of tracked object, etc.). The following settings must be configured in addition to the above HMD settings for developing Gear VR experiences using the plugin. For more information on developing Gear VR applications in Unreal Engine, refer to UE 4 documentations.
- [UE 4] For developing Android applications in Unreal Engine, make sure the project environment is set up for Android development: Android Quick Start Guide .
- [UE4] Follow the GearVR Quick Start Guide and configure the project for GearVR development.
- [UE4] Generate and include the OSIG file (oculus signature) in the directory. See: Oculus Signature File Generator.
- [Motive] Under the Data Streaming pane, set the Network Type setting under the Advanced Settings to Unicast. Note that the plugin currently only supports the Unicast broadcasting for streaming onto Android, multicasting will be supported in the future releases.
Connecting via Wifi
- 1. [Android] Connect the smartphone to the internet router which the host PC (server) is connected to.
- 2. [UE4] Configure the Client object.
- - Use Connection Type: Unicast.
- - Enable Auto Connect, unless there is a need to designate the network address.
- 3. [UE4] Connect a smartphone into the PC, build and run the project.
- 4. Make sure Motive is streaming, and the Gear VR HMD tracking data will be streamed into the UE4 application.
Connecting to a Designated Network.
HMD Tracking Setup
First of all, setup and optimize the motion capture volume as explained in the Getting Started guide or the Hardware Setup documentation. If you plan to install any obstacles (e.g. walls) within the capture volume, make sure they are non-reflective, and place and orient the cameras so that every corner is thoroughly captured by multiple cameras. For typical rigid body tracking, attach the rigid body markers as instructed in the Rigid Body Tracking page.
General Setup Steps
- Attach the markers on the HMD
- Create a rigid body asset
- Calibrate the Pivot Point of the rigid body
- Configure the rigid body settings in Motive
There are two different types of markers that can be used to track an object with OptiTrack systems: Passive markers and Active markers. Passive markers are retroreflective markers that reflect infrared light emitted from the IR LEDs on the camera. On the other hand, the active markers are LED markers that emit the IR light. Either type of marker can be used to track HMDs. For tracking with passive markers, the markers can be attached to the HMD using appropriate adhesives and marker posts. For tracking with active markers, the active CV1 clip can be used to easily attach 8 active markers onto the HMD. Depending on which marker you are using to track the HMD, the setup instructions will vary slightly.
Attachment & Placement
With the HMD bracket, simply attach the active marker bracket onto an CV1 HMD and use the active markers on it to define a rigid body.
When using the active markers, the markers are actively labeled individually from each of their own unique illumination patterns, and only the matching active markers get contributed to the rigid body solve. The biggest benefit from the individual labeling is that markers can be placed in perfectly symmetrical placements among multiple rigid body assets.
When attaching retroreflective markers, make sure markers are securely attached and readily captured by the cameras. For attaching the markers, we recommend using our 20 mm wide and 30 mm tall M4 threaded plastic marker bases with Acrylic adhesives, available at the webstore, to attach the markers onto the HMD.
A markered HMD will be defined as a rigid body in Motive. When placing markers, make sure the placement asymmetry is respected in the arrangement within the HMD. Also, the marker arrangements between multiple HMDs must be incongruent. For more details, read about marker placement from the Rigid Body Tracking page.
Unplug Oculus Positional Tracker
When using an OptiTrack motion capture system to track Oculus HMDs, avoid connecting its positional tracker, the Rift sensor, to the host PC. If the tracker is connected to the PC, IR LEDs on the connected HMD will illuminate, and it could interfere negatively with the IR tracking of the motion capture system. When Oculus tracker is not recognized by the computer, an error message may appear but you can ignore the error message (Critical Error - can't find the sensor) and proceed without the tracker.
Create an HMD Rigid Body
Creating Active Clip HMD
This feature can be used only with HMDs that have the OptiTrack Active HMD clips mounted.
For using OptiTrack system for VR applications, it is important that the pivot point of HMD rigid body gets placed at the appropriate location, which is at the root of the nose in between the eyes. When using the HMD clips, you can utilize the HMD creation tools in the Builder pane to have Motive estimate this spot and place the pivot point accordingly. It utilizes known marker configurations on the clip to precisely place the pivot point set the desired orientation.
HMDs with passive markers can utilize the External Pivot Alignment tool to calibrate the pivot point.
- First of all, make sure Motive is configured for tracking active markers.
- Open the Builder pane under View tab and click Rigid Bodies.
- Under the Type drop-down menu, select HMD. This will bring up the options for defining an HMD rigid body.
- Under the Orientation drop-down menu, select the desired orientation of the HMD. The orientation used for streaming to Unity is +Z forward and Unreal Engine is +X forward, or you can also specify the expected orientation axis on the client plugin side.
- Hold the HMD at the center of the tracking volume where all of the active markers are tracked well.
- Select the 8 active markers in the 3D viewport.
- Click Create. An HMD rigid body will be created from the selected markers and it will initiate the calibration process.
- During calibration, slowly rotate the HMD to collect data samples in different orientations.
- Once all necessary samples are collected, the calibrated HMD rigid body will be created.
Creating HMD from the markers
In Motive, select the markers that are attached to the HMD and create a rigid body (Hotkey: CTRL + T) as you would do for other rigid bodies. Once this is set up, use the External Pivot Alignment tool in the Builder pane (rigid body → Edit) to calibrate the pivot point of the created rigid body:
For using OptiTrack system for VR applications, it is important that the pivot point of HMD rigid body gets placed at the appropriate location, which is at the root of the nose in between the eyes. External Pivot Alignment tool utilizes both the OptiTrack mocap system and the HMD's own tracking system (Oculus Tracker / Vive Base Stations) for precisely calculating the location of the pivot point of an HMD. When using this feature, both systems must be tracking the HMD simultaneously.
HMD Compatibility Notes:
- First of all, connect the HMD Tracker (Oculus Tracker / Vive Base Stations) and have its corresponding software running in the background.
- Launch Motive.
- [Motive] Create a rigid body asset from the markers on the HMD.
- [Motive] Open the Builder pane and access editing options for rigid bodies.
- [Motive] Select the HMD rigid body in Motive.
- [Motive → Builder pane] Under Detected Type, it should indicate Custom RigidBody.
- [Motive → Builder pane] Set the desired number of sample counts (Default: 750) and the desired orientation.
- [Motive → Builder pane] Press start to initiate the sampling process.
- [Motive → Builder pane] During calibration, make sure both the mocap system and the HMD tracker (Oculus Tracker / Vive Base Stations) is tracking the HMD.
- [Motive → Builder pane] Once it starts collecting the samples, point the HMD towards the tracker and slowly rotate it and sample different orientations.
- [Motive → Builder pane] When it finishes collecting samples, it will display the offset distance, in mm, between positions of the pivot point detected by the HMD tracking system and the motion capture system.
- [Motive → Builder pane] If the results are satisfying, press Apply to calibrate the HMD rigid body.
Rigid Body Properties
Once the rigid body is created, select the asset and open the Properties pane. The following settings need to be configured for HMD tracking.
Deflection setting is the tolerable distance, in millimeters, that a rigid body marker may deviate from its expected position before it is unlabeled and unassociated with the rigid body. The deflection is set to 4 mm by default. For HMD tracking applications, we recommended lowering this value to 3 or 2. This will reduce the amount of computation required for labeling, and overall latency may be reduced.
- The tracking algorithm setting determines which protocol to use for solving the rigid body in Motive.
- Use the ray-based tracking algorithm for HMDs with passive markers. This algorithm is recommended because it allows tracking to be less susceptible to marker occlusions and solves rigid bodies more robustly without introducing trajectory gaps. Using ray-based solving will increase processing load, but since it can stabilize the tracking a lot, this is more suitable for tracking the passive marker HMDs where even minor occlusions can acutely affect the experience.
- Use the marker-based tracking algorithm for HMDs using active marker clips. Ray-based tracking is not supported for active markers since active markers themselves already provide stable tracking from active labeling using unique illumination patterns. Thus, marker-based tracking will provide robust tracking for HMDs with active markers.
The User Data value for each rigid body is used as an identification number to reference the rigid body in external applications. Log this value for each rigid body (including the HMD(s)) in the scene. This number will be used to associate the assets in the client applications using the plugin devices.
Min Marker Count (Passive Markers only)
This setting may need to be modified when tracking HMDs with passive markers. To prevent the swapping of the rigid body definition, set this setting to 4. When tracking multiple HMDs, there could be limitations to a variety of unique marker arrangements that could be achieved. If this value is set to a lower value, a set of three markers on an HMD may be congruent to another set in a different HMD, and the rigid body definitions may be switched in Motive.
Recalibrating Pivot Point
After the rigid body definition have been created for the HMD, the position and orientation of the rigid body pivot point should get placed precisely on the root of the player's nose with the respective orientation axis pointing the forward direction. If needed, you can use the Builder pane edit tools to recalibrate the HMD pivot point positions.
- The pivot point is placed on the bridge of the nose; specifically, right at the midpoint between two eyes.
- Orientation axes of the rigid body should precisely align with the physical HMD component.
Manually Calibrating the HMD Pivot Point
In most cases, you will be utilizing the previously described calibration tool for precisely positioning and orienting the HMD rigid body. However, if, for some reason, you want to manually position the pivot point without using the HMD Calibration feature, follow the instruction on this page: Manually Calibrating the HMD Pivot Point.
Export the Rigid Body
Tip: Once you have the rigid body asset for the HMD configured, you can export the asset into a MOTIVE file (*.motive) for future uses. Importing the MOTIVE file (e.g. CV1.tra) will load the rigid body (HMD) asset and make it available for use; however, the marker placement must remain unchanged in order to re-load previously created rigid bodies.
Once you have connected the client object and configured the script components on the assets from the scene. Play the scene, and if the properties are correctly configured, the assets in the game scene will be animated according to the position and orientation of the rigid bodies and the HMD in Motive. When playing the scene, the HMD and the assigned actor objects will be placed in respect to their location in Motive’s coordinate system disregards to their position in the editor.
Wireless Multiplayer Setup
When setting up multiplayer games on wireless clients, it is more beneficial for each client to make direct connection to both the tracking-server (Motive) and the game-server, rather than rebroadcasting the streamed tracking data through the game-server. Then, any of the game related actions that interacts with the tracking data can be processed on the game-server, and this server can send out the corresponding updates to the wireless clients. This allows the wireless clients to only receive both the tracking data or updates without having to send back any information; in other words, minimizing the number of data transfers needed. If wireless clients are sending data there will be a minimum of two transfers on the wireless network, and each transfer of data through wireless network is at risk of latency or lost packets.