OptiTrack Unity Plugin


Main pageOptiTrack Unity Plugin

Streaming tracking data into Unity

Plugin Overview

The OptiTrack Unity3D Plugin allows real-time streaming of live or recorded motion tracking data from Motive into Unity. Kinetic data from both rigid body and skeleton tracking can be streamed via the plugin, and received data can be used for animating objects and characters within a scene in Unity. The plugin contents are distributed in a unitypackage format and you can simply load this file into Unity projects. Once imported, the included C# scripts can be attached to Unity objects for instantiating a client and receiving the tracking data. Detailed steps will be explained in this guide.


Skeleton Retargetting

By integrating with Unity's animation system, Mecanim, the Unity3D plugin allows Motive to stream full body skeleton data without the need of separate retargeting middle software. The skeleton tracking data from Motive is streamed out as hierarchical bone segment orientations, and this data is fed into the Unity's Mecanim system which allows animating characters with different proportions.
Note: At the time of writing, Mecanim does not support explicit goals for inverse kinematics end-effectors when using real-time retargeting. In addition, you may observe a difference in the overall scale of the position data between the retargeted skeletal animations and streamed rigid bodies. These two limitations may lead to inconsistencies with actors interacting with rigid body props, and will hopefully be addressed in a future version of the integration.


Virtual Reality Experience

The plugin also features HMD tracking integration for virtual reality experiences. OptiTrack motion capture systems provide rigid body and full-body human motion tracking with outstanding tracking accuracy, and all of which can be accomplished with extremely low latency. Utilizing the tracking capability, an OptiTrack system can be used to provide fully immersive virtual reality experience by integrating a head mounted display (HMD) device. From this combined setup, even tiny movements can be precisely captured and reflected in the virtual space, and such accurate representation will deliver an immersive virtual experience.
For general instructions on developing VR application in Unity, refer to the Unity documentation: Unity Virtual Reality tutorial.
  • Supported HMD Models: Oculus DK2, Oculus CV1, and Gear VR.


Versions Requirements

  • Unity Version: 2017.2 / 2017.1 / 5.4 or above.
  • The version of the Oculus SDK should be compatible with Unity version. See: Unity-SDK version compatibility


Motive Setup (Server)[edit]


Streaming Setup[edit]

Data Streaming settings in Motive.
Tracking data can be streamed in real-time (Live Mode) or from a recorded capture data (Edit Mode). From Motive, data broadcasting can be enabled, or disabled, from the Data Streaming pane, and in this panel, you can configure the connection settings and decide which data to include within the delivered data packet. For the best performance, disable streaming of unnecessary data types. This will reduce the size of data packets and simplify the parsing process on the client side. However, make sure the following data types are enabled for streaming:

Streaming Rigid Bodies or HMD[edit]

  • Stream Rigid Bodies → True

Streaming Skeletons[edit]

  • Stream Local Rigid Bodies → True
  • Stream Skeleton → True
  • Skeleton As Rigid Bodies → False
  • Bone Naming Convention → FBX.
This set the bone names to match the convention used in exported FBX files.

Streaming Over Wifi Network[edit]

  • Transmission Type → Unicast

Unity Setup (Client)[edit]


Unity plugin files.

Import the Plugin Package[edit]

While in the Unity project, double click on the plugin unitypackage file and import the plugin assets into the project. When the package has been successfully imported, the followign contents will be available within the project:
Plugin Contents
Folder Content Description
Assets/OptiTrack All of the Unity plugin contents are included in this folder.
Assets/OptiTrack/Scripts This is the folder that you will mainly use. It contains plugin C# script components that can be imported into Unity objects for receiving streamed data.
Assets/OptiTrack/Editor This folder contains editor scripts and contents for visualization of the rigid body markers.
Assets/OptiTrack/Plugins This folder contains the plugin libraries and header files.
Assets/OptiTrack/Prefabs This folder contains prefabs of sample client object, skeleton object, and HMD object.
Assets/OptiTrack/Scenes This folder contains sample Unity scene that includes pre-configured client, rigid body, skeleton, and HMD objects.


Setting Up the Client Object[edit]

In order to receive tracking data from a server application (e.g. Motive), you need to create a client object. A client object can be any object within a scene in Unity, and it can be designated by attaching OptitrackStreamingClient.cs script. This script receives the tracking data from the connected server application (e.g. Motive) and makes the data available within the scene. You can either attach the client script onto an existing object or an empty object. Also, you can just import the sample from the Assets/Optitrack/Prefabs folder.
  • [Motive] In the Data Streaming pane, configure the desired connection settings and check the Broadcast Frame Data box.
  • [Unity] Under the Prefabs folder, import the client prefab object into the scene, or you can just attach OptitrackStreamingClient.cs script onto an existing object.
  • [Unity] In the Client object, configure the connection settings from the OptiTrack Streaming Client script so that the values (Connection Type, Local Address, Server Address, Server Command Port, Server Data Port) match the parameters under the Data Streaming pane in Motive.
  • Local Address: Local IP Address of the PC that the client application is running on.
  • Server Address: IP address of the PC that the server application is running on.
  • [Unity] Set the Bone Naming Convention type so that it matches the convention set in Motive.
  • [Unity] If you wish to receive tracking data from more than one server instances, you may create multiple objects with the client script attached.


Client object in Unity and the corresponding Motive data streaming network settings.

Info2.png

Position data in Unity

Position component of the streamed data will be received as reference positions in Unity. In other words, the streamed position will reflect object's relative displacement from its root object or the global origin if the object is placed at the root level of the hierarchy in Unity. For this reason, although it is not strictly necessary, you may find it helpful to organize your tracked objects as children of a common game object (e.g. the Client object) that can then be re-positioned in the Unity scene to effectively determine the location of the tracking volume origin. And doing so will make it easier to adjust the global origin of Motive within a scene in Unity.
[Unity Hierarchy Pane] Components set to children of the root Client object.

Animating Rigid Body[edit]

  1. [Unity] On an object that you wish to animate, attach OpitrackRigidBody.cs script.
  2. [Unity] In the Streaming Client entry, link the Client object which the client script is attached to. By default, it searches for an existing client instance, but this must be specified when there are more than one streaming client objects.
  3. [Unity] For the Rigid Body ID entry, enter the User Data ID which is assigned to each rigid body asset in Motive.
  4. [Motive] Make sure Motive is streaming, play the scene in Motive.
  5. [Unity] Play the scene. The linked object will be animated according to the rigid body movement in Motive.
OptiTrack Rigid Body configuration along with the rigid body properties in Motive. Configured Streaming ID must match the Rigid Body ID designated from the client side.

Animating Skeleton[edit]

  1. [Unity] On Unity characters, attach OptitrackSkeletonAnimator.cs script as one of its components.
  2. [Unity] For the Streaming Client entry, link the object which the client script is attached to. By default, it searches for an existing client instance, but this must be specified when there are more than one streaming client objects.
  3. [Unity] Enter Skeleton Asset Name which is Assigned in Motive
  4. [Unity] For the Destination Avatar entry, link to the character that the script is loaded to.
  5. [Motive] From the Data Streaming pane, make sure Motive is streaming.
  6. [Unity] Play the scene. When everything is set up properly, the linked avatar in Unity will be animated according to the streamed skeleton in Motive. The position of the actor will be in its reference position as explained above.
Optitrack Skeleton Animator script configuration from a character in Unity.

Streaming Marker Positions[edit]

  1. [Unity] On the OptiTrack Streaming instance, enable Draw Markers setting.
  2. [Motive] In the Data Streaming pane, enable streaming of the markers. Here, you can choose to stream labeled markers, unlabeled markers, or both.
  3. [Unity] Make sure the streaming setting is set up correctly, and play the scene.
  4. [Unity] Each marker will be drawn as sphere objects in the scene, as shown in the screenshot below.
Skeleton labeled markers drawn in Unity scene.

Integrating HMD in Unity[edit]


Oculus DK2, CV1[edit]

  1. [Motive] First of all, follow the HMD Tracking Setup section and prepare HMD rigid bodies in Motive.
  2. Make sure the HMD software is running.
  3. [Unity] In the Unity project, check the Virtual Reality Supported box from the player settings (Edit → Project Settings → Player), as shown in the image below.
  4. [Unity] From the Prefabs folder (OptiTrack/Prefabs) of the imported plugin package, import the HMD prefab (HMD - OptiTrack) object into the scene.
  5. [Unity] In the Streaming Client entry, specify the object which the streaming client is attached to. By default, it searches for an existing client instance, but this must be specified when there are more than one streaming client objects.
  6. [Unity] For the Rigid Body ID entry, enter the User Data ID value given for the HMD rigid body in Motive.
  7. [Unity] For the Rigid Body Orientation entry, enter the orientation of the rigid body coordinate axis that is defined in Motive.
  8. [Unity] Play the scene. When the HMD is properly connected, the scene will be visible from the HMD.
  • HMD object and its components in Unity
  • VR Supported setting in Unity.

Info2.png

OVRManager

Having OVRManager enabled within the project may introduce an additional offset to the position of the HMD. If you are seeing positional offset in the HMD position, remove the OVRManager object.

Android: Gear VR[edit]

Gear VR devices can also be integrated and the tracking information can be streamed via wifi connections using a router with sufficient bandwidth. The required bandwidth will vary depending on many factors (e.g. router type, the number of tracked object, etc.). For more specific information on this setup, contact us. The following settings must be configured in addition to the above HMD settings for developing Gear VR experiences using the plugin. For more information on developing Gear VR applications in Unity, refer to Unity documentations.

Settings[edit]

1. [Unity] For developing Android applications in Unity, make sure the environment is set up for Android development: Getting Started with Android Development. Below are some of the important steps:

  • Download the Unity Android package under File → Build Settings... → Android.
  • Put your phone into the developer mode. For example, on Samsung Galaxy, go to Settings → About Phone and tap on the build number multiple times to reveal the developer options.
  • Install and enable the developer mode by turning on the Gear VR Service → Memory → Developer Mode.


2. [Unity] Include the OSIG file (oculus signature) in the Project/Assets/Plugins/Android/assets/ directory. See: Oculus Signature File Generator.

UnityPlugin OSIG.png

3. [Unity] Open the Player Settings (Edit → Project Settings → Player) and check the Virtual Reality Supported box under the XR Settings section. Then add the Oculus SDK under the list of Virtual Reality SDK as shown in the image below.

UnityPlugin VRSettings.png

4. [Unity] In Player Settings under the Other Settings section, there are a few other options that need to be set:

  • Configuration → Internet Access → Require
  • Identification → Package Name → com.[Company Name].[Product Name]. This just needs to be unique.
  • Identification → Minimum API Level → (API level 19).
  • UnityPlugin AndroidSettings1.png
  • UnityPlugin AndroidSettings2.png


5. [Motive] Under the Data Streaming pane, set the Network Type setting under the Advanced Settings to Unicast.

Info2.png

Note: the plugin currently only supports the Unicast broadcasting for streaming onto Android.

Connecting via Wifi[edit]

1. [Motive] Set the streaming IP address to the IP address of the router.

2. [Android] Connect the smartphone to the router which the Motive is streaming to.

3. [Unity] Configure the Client object. This can be done using the Client - OptiTrack prefab included in the plugin.

  • Set the Connection Type to Unicast.
  • Set the Local Address to the IP address of the phone. The local address can usually be found under the phone's settings. The IP address for the Samsung Galaxy, for example, can be found under Settings → About Phone → Status → IP Address". For some phones, the address of 0.0.0.0 may work.
  • Set the Server Address to the router's IP address, or whatever the IP address that the host PC (server) is streaming data to. This address can also be found under the phone's settings when the smartphone is connected to the wifi router of the tracking server.
UnityPlugin ConnectWifi.png

4. [Unity] Configure the HMD object.

  • Drag the “HMD - OptiTrack” prefab into the scene.
  • Drag your Client object into the Streaming Client property.
  • Set the Rigid Body ID to the streaming ID of whatever rigid body you are using as the HMD in Motive.

5. [Unity] Connect a smartphone into the PC, build and run the project. Make sure network permission is given to the Unity application.

6. [Motive] Double check that Motive is streaming. If everything is set up properly, Motive will provide the tracking data to the phone.


HMD Tracking Setup[edit]


First of all, setup and optimize the motion capture volume as explained in the Getting Started guide or the Hardware Setup documentation. If you plan to install any obstacles (e.g. walls) within the capture volume, make sure they are non-reflective, and place and orient the cameras so that every corner is thoroughly captured by multiple cameras. For typical rigid body tracking, attach the rigid body markers as instructed in the Rigid Body Tracking page.

General Setup Steps

  1. Attach the markers on the HMD
  2. Create a rigid body asset
  3. Calibrate the Pivot Point of the rigid body
  4. Configure the rigid body settings in Motive

Marker Attachment[edit]

There are two different types of markers that can be used to track an object with OptiTrack systems: Passive markers and Active markers. Passive markers are retroreflective markers that reflect infrared light emitted from the IR LEDs on the camera. On the other hand, the active markers are LED markers that emit the IR light. Either type of marker can be used to track HMDs. For tracking with passive markers, the markers can be attached to the HMD using appropriate adhesives and marker posts. For tracking with active markers, the active CV1 clip can be used to easily attach 8 active markers onto the HMD. Depending on which marker you are using to track the HMD, the setup instructions will vary slightly.

Active Markers[edit]

Attachment & Placement[edit]

With the HMD bracket, simply attach the active marker bracket onto an CV1 HMD and use the active markers on it to define a rigid body.

When using the active markers, the markers are actively labeled individually from each of their own unique illumination patterns, and only the matching active markers get contributed to the rigid body solve. The biggest benefit from the individual labeling is that markers can be placed in perfectly symmetrical placements among multiple rigid body assets.

  • HMD markers with passive markers
  • Putting the HMD bracket on
  • When detaching the bracket from the HMD, first pull up the two latches behind the bracket and slowly pull it from the top.

Passive Markers[edit]

Attachment[edit]

HMD markers with passive markers

When attaching retroreflective markers, make sure markers are securely attached and readily captured by the cameras. For attaching the markers, we recommend using our 20 mm wide and 30 mm tall M4 threaded plastic marker bases with Acrylic adhesives, available at the webstore, to attach the markers onto the HMD.

Placement[edit]

A markered HMD will be defined as a rigid body in Motive. When placing markers, make sure the placement asymmetry is respected in the arrangement within the HMD. Also, the marker arrangements between multiple HMDs must be incongruent. For more details, read about marker placement from the Rigid Body Tracking page.

Unplug Oculus Positional Tracker[edit]

When using an OptiTrack motion capture system to track Oculus HMDs, avoid connecting its positional tracker, the Rift sensor, to the host PC. If the tracker is connected to the PC, IR LEDs on the connected HMD will illuminate, and it could interfere negatively with the IR tracking of the motion capture system. When Oculus tracker is not recognized by the computer, an error message may appear but you can ignore the error message (Critical Error - can't find the sensor) and proceed without the tracker.

Create an HMD Rigid Body[edit]

Creating Active Clip HMD

Creating an HMD rigid body in the Builder pane.

This feature can be used only with HMDs that have the OptiTrack Active HMD clips mounted.

For using OptiTrack system for VR applications, it is important that the pivot point of HMD rigid body gets placed at the appropriate location, which is at the root of the nose in between the eyes. When using the HMD clips, you can utilize the HMD creation tools in the Builder pane to have Motive estimate this spot and place the pivot point accordingly. It utilizes known marker configurations on the clip to precisely place the pivot point set the desired orientation.

Info2.png

HMDs with passive markers can utilize the External Pivot Alignment tool to calibrate the pivot point.

Steps[edit]

  1. First of all, make sure Motive is configured for tracking active markers.
  2. Open the Builder pane under View tab and click Rigid Bodies.
  3. Under the Type drop-down menu, select HMD. This will bring up the options for defining an HMD rigid body.
  4. Under the Orientation drop-down menu, select the desired orientation of the HMD. For streaming to Unity, the default orientation is +Z forward, and for streaming to Unreal Engine the default orientation is +X forward. You can also specify the expected orientation axis on the client plugin side.
  5. Hold the HMD at the center of the tracking volume where all of the active markers are tracked well.
  6. Select the 8 active markers in the 3D viewport.
  7. Click Create. An HMD rigid body will be created from the selected markers and it will initiate the calibration process.
  8. During calibration, slowly rotate the HMD to collect data samples in different orientations.
  9. Once all necessary samples are collected, the calibrated HMD rigid body will be created.


Creating Passive Marker HMD

In Motive, select the markers that are attached on the HMD and create a rigid body (Hotkey: CTRL + T) as you would do for other rigid bodies. Once this is setup, use the External Pivot Alignment tool in the Builder pane (rigid body → Edit) to calibrate the pivot point of the created rigid body:

This tool can be used for HMD with passive markers.

For using OptiTrack system for VR applications, it is important that the pivot point of HMD rigid body gets placed at the appropriate location, which is at the root of the nose in between the eyes. External Pivot Alignment tool utilizes both the OptiTrack mocap system and the HMD's own tracking system (Oculus Tracker / Vive Base Stations) for precisely calculating the location of the pivot point of an HMD. When using this feature, both systems must be tracking the HMD simultaneously.

Info2.png

HMD Compatibility Notes:

  • For Motive versions 2.1 Final and later: Oculus and Vive HMDs are supported.
  • For older versions of Motive: Only Oculus HMDs are supported.
  • The corresponding native HMD tracker must be tracking the HMD along with the motion capture system simultaneously.

Steps[edit]

  1. Connect the Oculus Tracker and have the Oculus software running in the background.
  2. Launch Motive.
  3. [Motive] Create a rigid body asset from the markers on the HMD.
  4. [Motive] Open the Builder pane and access editing options for rigid bodies.
  5. [Motive] Select the HMD rigid body in Motive.
  6. [Motive → Builder pane] Under Detected Type, it should indicate Custom RigidBody.
  7. [Motive → Builder pane] Set the desired number of sample counts (Default: 750) and the desired orientation.
  8. [Motive → Builder pane] Press start to initiate the sampling process.
  9. [Motive → Builder pane] During calibration, make sure both the mocap system and the HMD tracker (Oculus Tracker / Vive Base Stations) is tracking the HMD.
  10. [Motive → Builder pane] Once it starts collecting the samples, point the HMD towards the tracker and slowly rotate it and sample different orientations.
  11. [Motive → Builder pane] When it finishes collecting samples, it will display the offset distance, in mm, between positions of the pivot point detected by the HMD tracking system and the motion capture system.
  12. [Motive → Builder pane] If the results are satisfying, press Apply to calibrate the HMD rigid body.


Rigid Body Properties[edit]

Once the rigid body is created, select the asset and open the Properties pane. The following settings need to be configured for HMD tracking.

Rigid body properties for the HMD asset.

Deflection[edit]

Deflection setting is the tolerable distance, in millimeters, that a rigid body marker may deviate from its expected position before it is unlabeled and unassociated with the rigid body. The deflection is set to 4 mm by default. For HMD tracking applications, we recommended lowering this value to 3 or 2. This will reduce the amount of computation required for labeling, and overall latency may be reduced.

Tracking Algorithm[edit]

The tracking algorithm setting determines which protocol to use for solving the rigid body in Motive.
Passive Markers[edit]
Use the ray-based tracking algorithm for HMDs with passive markers. This algorithm is recommended because it allows tracking to be less susceptible to marker occlusions and solves rigid bodies more robustly without introducing trajectory gaps. Using ray-based solving will increase processing load, but since it can stabilize the tracking a lot, this is more suitable for tracking the passive marker HMDs where even minor occlusions can acutely affect the experience.
Active Markers[edit]
Use the marker-based tracking algorithm for HMDs using active marker clips. Ray-based tracking is not supported for active markers since active markers themselves already provide stable tracking from active labeling using unique illumination patterns. Thus, marker-based tracking will provide robust tracking for HMDs with active markers.

Streaming ID[edit]

The User Data value for each rigid body is used as an identification number to reference the rigid body in external applications. Log this value for each rigid body (including the HMD(s)) in the scene. This number will be used to associate the assets in the client applications using the plugin devices.

Min Marker Count (Passive Markers only)[edit]

This setting may need to be modified when tracking HMDs with passive markers. To prevent the swapping of the rigid body definition, set this setting to 4. When tracking multiple HMDs, there could be limitations to a variety of unique marker arrangements that could be achieved. If this value is set to a lower value, a set of three markers on an HMD may be congruent to another set in a different HMD, and the rigid body definitions may be switched in Motive.

Recalibrating Pivot Point[edit]

After the rigid body definition have been created for the HMD, the position and orientation of the rigid body pivot point should get placed precisely on the root of the player's nose with the respective orientation axis pointing the forward direction. If needed, you can use the Builder pane edit tools to recalibrate the HMD pivot point positions.

  • The pivot point is placed on the bridge of the nose; specifically, right at the midpoint between two eyes.
  • Orientation axes of the rigid body should precisely align with the physical HMD component.
  • Uncalibrated HMD rigid body definition. The pivot point is not positioned nor oriented properly.
  • Calibrated HMD rigid body definition. x-axis forward for UE4 integration.

Manually Calibrating the HMD Pivot Point[edit]

In most cases, you will be utilizing the previously described calibration tool for precisely positioning and orienting the HMD rigid body. However, if, for some reason, you want to manually position the pivot point without using the HMD Calibration feature, follow the instruction on this page: Manually Calibrating the HMD Pivot Point.

  • Buidler pane HMD calibration tab.
  • HMD calibration results

Export the Rigid Body[edit]

Info2.png

Tip: Once you have the rigid body asset for the HMD configured, you can export the asset into a TRA file for future uses. Importing the TRA file (e.g. CV1.tra) will load the rigid body (HMD) asset and make it available for use; however, the marker placement must remain unchanged in order to re-load previously created rigid bodies.

Live Streaming[edit]


Once you have connected the client object and configured the script components on the assets from the scene. Play the scene, and if the properties are correctly configured, the assets in the game scene will be animated according to the position and orientation of the rigid bodies and the HMD in Motive. When playing the scene, the HMD and the assigned actor objects will be placed in respect to their location in Motive’s coordinate system disregards to their position in the editor.

Wireless Multiplayer Setup[edit]


When setting up multiplayer games with wireless clients, it is more beneficial for each client to make direct connection to both the tracking-server (Motive) and the game-server, rather than rebroadcasting the streamed tracking data through the game-server. Then, any of the game related actions that interacts with the tracking data can be processed on the game-server, and this server can send out the corresponding updates to the wireless clients. This allows the wireless clients to only receive both the tracking data or updates without having to send back any information; in other words, minimizing the number of data transfers needed. If wireless clients are sending data there will be a minimum of two transfers on the wireless network, and each transfer of data through wireless network is at risk of latency or lost packets.

WirelessClientSetup.png