Difference between revisions of "OptiTrack Unity Plugin"

Line 135: Line 135:
 
[[Image:MarkerObjectUnity_21.png|thumb|800 px|center|Skeleton labeled markers drawn in Unity scene.]]
 
[[Image:MarkerObjectUnity_21.png|thumb|800 px|center|Skeleton labeled markers drawn in Unity scene.]]
  
=Integrating HMD in Unity=
+
=Integrating HMDs=
 
----
 
----
The plugin also features HMD tracking integration for virtual reality experiences. For general instructions on developing VR application in Unity, refer to the Unity documentation: [https://unity3d.com/learn/tutorials/topics/virtual-reality Unity Virtual Reality tutorial].
 
  
{{Info|Oculus DK2, CV1, and Gear VR HMD models are supported}}
+
OptiTrack motion capture systems can be used to track head mounted displays (HMD) and integrate the tracking data into Unity for unique VR applications. For instructions on integrating HMD tracking data into Unreal Engine, please refer to the corresponding pages:
  
<tabs>
+
* [[Unity: Oculus HMD Tracking]]
<tab name="Oculus" style="padding:0 1em 0 1em;">
+
* [[Unity: Vive HMD Tracking]]
==Oculus DK2, CV1==
 
# '''[Motive]''' First of all, follow the [[#HMD Tracking Setup|HMD Tracking Setup]] section and prepare HMD rigid bodies in Motive.
 
# Make sure the HMD software is running.
 
# '''[Unity]''' In the Unity project, check the '''Virtual Reality Supported''' box from the player settings (Edit &rarr; Project Settings &rarr; Player), as shown in the image below.
 
# '''[Unity]''' From the Prefabs folder (OptiTrack/Prefabs) of the imported plugin package, import the HMD prefab (HMD - OptiTrack) object into the scene.
 
# '''[Unity]''' In the '''Streaming Client''' entry, specify the object which the streaming client is attached to. By default, it searches for an existing client instance, but this must be specified when there are more than one streaming client objects.
 
# '''[Unity]''' For the '''Rigid Body ID''' entry, enter the User Data ID value given for the HMD rigid body in Motive.
 
# '''[Unity]''' For the '''Rigid Body Orientation''' entry, enter the orientation of the rigid body coordinate axis that is defined in Motive.
 
# '''[Unity]'''  Play the scene. When the HMD is properly connected, the scene will be visible from the HMD.
 
  
<center><ul>
+
{{Info|
<li class="thumblist">[[Image:Unity_HMD.png|thumb|center|HMD object and its components in Unity]]</li>
+
====Supported HMDs====
<li class="thumblist">[[Image:UnityVRSupported.png|thumb|center|480 px|VR Supported setting in Unity.]]</li>
+
{{Indent|At the time of writing, the following HMDs are supported:
</ul></center>
+
*HTC Vive
 
+
*HTC Vive Pro
{{Indent|{{Info|'''OVRManager'''
+
*Oculus DK2
 
+
*Oculus CV1
:Having OVRManager enabled within the project may introduce an additional offset to the position of the HMD. If you are seeing positional offset in the HMD position, remove the OVRManager object.
+
*Oculus GearVR
 
}}}}
 
}}}}
</tab>
 
<tab name="Android: GearVR" style="padding:0 1em 0 1em;">
 
==Android: Gear VR==
 
<div class="padded">
 
Gear VR devices can also be integrated and the tracking information can be streamed via wifi connections using a router with sufficient bandwidth. The required bandwidth will vary depending on many factors (e.g. router type, the number of tracked object, etc.). For more specific information on this setup, [http://optitrack.com/contact/ contact us]. The following settings must be configured in addition to the above HMD settings for developing Gear VR experiences using the plugin. For more information on developing Gear VR applications in Unity, refer to Unity documentation.
 
 
====Settings====
 
<div class="padded">
 
'''1. [Unity]''' For developing Android applications in Unity, make sure the environment is set up for Android development: [https://docs.unity3d.com/Manual/android-GettingStarted.html Getting Started with Android Development]. Below are some of the important steps:
 
<div class="padded">
 
* Download the Unity Android package under ''File &rarr; Build Settings... &rarr; Android''.
 
* Put your phone into the developer mode. For example, on Samsung Galaxy, go to ''Settings &rarr; About Phone'' and tap on the build number multiple times to reveal the developer options.
 
* Install and enable the developer mode by turning on the ''Gear VR Service &rarr; Memory &rarr; Developer Mode''.
 
</div>
 
 
 
'''2. [Unity]''' Include the OSIG file (oculus signature) in the <code>Project/Assets/Plugins/Android/assets/</code> directory. See: [https://developer.oculus.com/osig/ Oculus Signature File Generator].
 
 
[[Image:UnityPlugin_OSIG.png|center]]
 
 
'''3. [Unity]''' Open the Player Settings (Edit &rarr; Project Settings &rarr; Player) and check the '''Virtual Reality Supported''' box under the '''XR Settings''' section. Then add the Oculus SDK under the list of Virtual Reality SDK as shown in the image below.
 
 
[[Image:UnityPlugin_VRSettings.png|center]]
 
 
'''4. [Unity]''' In Player Settings under the '''Other Settings''' section, there are a few other options that need to be set:
 
<div class="padded">
 
* Configuration &rarr; Internet Access &rarr; Require
 
* Identification &rarr; Package Name &rarr; com.[Company Name].[Product Name]. This just needs to be unique.
 
* Identification &rarr; Minimum API Level &rarr; (API level 19).
 
</div>
 
 
<center><ul>
 
<li class="thumblist">[[Image:UnityPlugin_AndroidSettings1.png]]</li>
 
<li class="thumblist">[[Image:UnityPlugin_AndroidSettings2.png]]</li>
 
</ul></center>
 
 
 
'''5. [Motive]''' Under the [[Data Streaming pane]], set the Network Type setting under the Advanced Settings to ''Unicast''.
 
 
{{Indent|{{Info2|'''Note:''' the plugin currently only supports the Unicast broadcasting for streaming onto Android.}}}}
 
</div>
 
 
====Connecting via Wifi====
 
<div class="padded">
 
'''1. [Motive]''' Set the streaming IP address to the IP address of the router.
 
 
'''2. [Android]''' Connect the smartphone to the router which the Motive is streaming to.
 
 
'''3. [Unity]''' Configure the Client object.  This can be done using the ''Client - OptiTrack'' prefab included in the plugin.
 
:* Set the '''Connection Type''' to Unicast.
 
:* Set the '''Local Address''' to the IP address of the phone. The local address can usually be found under the phone's settings. The IP address for the Samsung Galaxy, for example, can be found under ''Settings &rarr; About Phone &rarr; Status &rarr; IP Address". For some phones, the address of 0.0.0.0 may work.
 
:* Set the '''Server Address''' to the router's IP address, or whatever the IP address that the host PC (server) is streaming data to. This address can also be found under the phone's settings when the smartphone is connected to the wifi router of the tracking server.
 
 
[[Image:UnityPlugin_ConnectWifi.png|center]]
 
 
'''4. [Unity]''' Configure the HMD object.
 
:* Drag the “HMD - OptiTrack” prefab into the scene.
 
:* Drag your Client object into the Streaming Client property.
 
:* Set the Rigid Body ID to the streaming ID of whatever rigid body you are using as the HMD in Motive.
 
 
'''5. [Unity]''' Connect a smartphone into the PC, build and run the project. Make sure network permission is given to the Unity application.
 
 
'''6. [Motive]''' Double check that Motive is streaming. If everything is set up properly, Motive will provide the tracking data to the phone.
 
</tab>
 
</tabs>
 
 
 
{{HMD Setup}}
 
 
<center><ul>
 
<li class="thumblist">[[Image:HMDSetup_Active1_Unity.png|thumb|Buidler pane HMD calibration tab.]]</li>
 
<li class="thumblist">[[Image:HMDSetup_Active2.png|thumb|HMD calibration results]]</li>
 
</ul></center>
 
 
==Export the Rigid Body==
 
<div class="padded">
 
{{Info|'''Tip:''' Once you have the rigid body asset for the HMD configured, you can export the asset into a TRA file for future uses. Importing the TRA file (e.g. CV1.tra) will load the rigid body (HMD) asset and make it available for use; however, the marker placement '''must''' remain unchanged in order to re-load previously created rigid bodies.}}
 
</div>
 
  
 
=Live Streaming=
 
=Live Streaming=

Revision as of 16:09, 15 July 2019

Main pageOptiTrack Unity Plugin

Streaming tracking data into Unity

Plugin Overview

The OptiTrack Unity3D Plugin allows you to stream real-time rigid body, skeleton, and HMD tracking data from Motive into Unity. Using the streamed data, objects and characters in the scene can be animated. The plugin contents are distributed in the unitypackage format and you can simply load this file into Unity projects to import its contents. Once imported, included C# scripts can be attached to Unity objects for instantiating a client for receiving the tracking data. This article focuses on how to set up and use the plugin.

Versions Requirements

  • Unity Version: 2017.2 / 2017.1 / 5.4 or above.
  • Oculus SDK should be compatible with Unity version.

Info2.png

Notes on HMD Compatibility

  • As of OptiTrack Unreal Engine 4 Plugin 1.2.1, the OptiTrack Oculus plugin supports Oculus DK2, CV1, and Gear VR HMD models.
  • Vive HMDs can be integrated through the OptiTrack OpenVR Driver.

Motive Setup (Server)


Streaming Setup

Data Streaming settings in Motive.

In Motive, the tracking data can be streamed in real-time either from a live capture (Live Mode) or recorded capture (Edit Mode). The streaming settings are configured in the Data Streaming pane, and in this pane, the Broadcast Frame Data must be enabled to start streaming the data to a local network interface designated by the IP address in the Local Interface section.

For the best performance, disable streaming of unnecessary data types is recommended. This will reduce the size of data packets and simplify the parsing process on the client side. Please make sure the appropriate data types are enabled, in Motive, for streaming:

Streaming in Motive

Open the Data Streaming pane in Motive and configure the settings below:

  • Enable Broadcast Frame Data setting at the top
  • From Local Interface drop-down menu, select a network address that you wish to stream the data over to.
  • Select desired data types to stream under streaming options. Streaming of rigid body is required. Streaming of skeleton is optional for skeleton tracking. For tracking HMDs, disabling the Marker streaming is adviced.
  • Set the Skeleton Coordinates to Local.
  • When streaming skeletons, set the appropriate bone naming convention for the client application. (If the character uses the FBX naming convention, this will need to be set to FBX.)
  • For streaming over a wifi network, setting the Transmission Type to Unicast is strongly adviced.

Info2.png

Additional Tips

  • In order to stream data from Edit mode, a capture recording must be playing back in Motive.
  • For best results, it is advised to run Motive and Unreal Engine separately on different computers, so that they are not competing for processing resources.
  • When streaming the data over a wifi network, Unicast transmission must be used.

Unity Setup (Client)


Unity plugin files.

Import Plugin Package

While in the Unity project, double-click on the plugin unitypackage file and import the plugin assets into the project. When the package has been successfully imported, the following contents will be available within the project:

Plugin Contents
Folder Content Description
Assets/OptiTrack All of the Unity plugin contents are included in this folder.
Assets/OptiTrack/Scripts This is the folder that you will mainly use. It contains plugin C# script components that can be imported into Unity objects for receiving streamed data.
Assets/OptiTrack/Editor This folder contains editor scripts and contents for visualization of the rigid body markers.
Assets/OptiTrack/Plugins This folder contains the plugin libraries and header files.
Assets/OptiTrack/Prefabs This folder contains prefabs of sample client object, skeleton object, and HMD object.
Assets/OptiTrack/Scenes This folder contains sample Unity scene that includes pre-configured client, rigid body, skeleton, and HMD objects.

Setting Up the Client Object

In order to receive tracking data from a server application (e.g. Motive), you need to create a client object. A client object can be any object within a scene in Unity, and it can be assigned by attaching OptitrackStreamingClient.cs script. This script receives the tracking data from the connected server application (e.g. Motive) and makes the data available within the scene. You can either attach the client script onto an existing object or an empty object. Also, you can just import the sample from the Assets/Optitrack/Prefabs folder.

  • [Motive] In the Data Streaming pane, configure the desired connection settings and check the Broadcast Frame Data box.
  • [Unity] Under the Prefabs folder, import the client prefab object into the scene, or you can just attach OptitrackStreamingClient.cs script onto an existing object.
  • [Unity] In the Client object, configure the connection settings from the OptiTrack Streaming Client script so that the values (Connection Type, Local Address, Server Address, Server Command Port, Server Data Port) match the parameters under the Data Streaming pane in Motive.
  • Local Address: Local IP Address of the PC that the client application is running on.
  • Server Address: IP address of the PC that the server application is running on.
  • [Unity] Set the Bone Naming Convention type so that it matches the convention set in Motive.
  • [Unity] If you wish to receive tracking data from more than one server instances, you may create multiple objects with the client script attached.


Client object in Unity and the corresponding Motive data streaming network settings.

Info2.png

Position Data in Unity

Position component of the streamed data will be received as reference positions in Unity. In other words, the streamed position will reflect the object's relative displacement from its root object or the global origin if the object is placed at the root level of the hierarchy in Unity. For this reason, although it is not strictly necessary, you may find it helpful to organize your tracked objects as children of a common game object (e.g. the Client object) that can then be re-positioned in the Unity scene to effectively determine the location of the tracking volume origin. And doing so will make it easier to adjust the global origin of Motive within a scene in Unity.
[Unity Hierarchy Pane] Components set to children of the root Client object.

Animating Rigid Body

  1. [Unity] On an object that you wish to animate, attach the OpitrackRigidBody.cs script.
  2. [Unity] In the Streaming Client entry, link the Client object which the OptitrackStreamingClient.cs is attached to. By default, it searches for an existing client instance, however, this must be specified when there are more than one streaming client objects in the scene.
  3. [Unity] For the Rigid Body ID entry, input the streaming ID of corresponding rigid body asset in Motive. The streaming ID can be found, and changed, under the rigid body properties.
  4. [Motive] Make sure Motive is tracking and streaming the data.
  5. [Unity] Play the scene. The linked object will be animated according to the associated rigid body movement in Motive.
OptiTrack Rigid Body configuration along with the rigid body properties in Motive. Configured Streaming ID must match the Rigid Body ID designated from the client side.

Animating Skeleton

By integrating with Unity's animation system, Mecanim, the Unity3D plugin allows Motive to stream full body skeleton data. The skeleton tracking data from Motive is streamed out as hierarchical bone segment orientations, and this data is fed into the Unity's Mecanim system which allows animating characters with different proportions.

Warning2.png

Note: At the time of writing, Mecanim does not support explicit goals for inverse kinematics end-effectors when using real-time retargeting. In addition, you may observe a difference in the overall scale of the position data between the retargeted skeletal animations and streamed rigid bodies. These two limitations may lead to inconsistencies with actors interacting with rigid body props, and will hopefully be addressed in a future version of the integration.

Steps

  1. [Unity] On Unity characters, attach OptitrackSkeletonAnimator.cs script as one of its components.
  2. [Unity] For the Streaming Client entry, link the object which the client script is attached to. By default, it searches for an existing client instance, but this must be specified when there are more than one streaming client objects.
  3. [Unity] Enter Skeleton Asset Name which is Assigned in Motive
  4. [Unity] For the Destination Avatar entry, link to the character that the script is loaded to.
  5. [Motive] From the Data Streaming pane, make sure Motive is streaming.
  6. [Unity] Play the scene. When everything is set up properly, the linked avatar in Unity will be animated according to the streamed skeleton in Motive. The position of the actor will be in its reference position as explained above.
Optitrack Skeleton Animator script configuration from a character in Unity.

Streaming Marker Positions

  1. [Unity] On the OptiTrack Streaming instance, enable Draw Markers setting.
  2. [Motive] In the Data Streaming pane, enable streaming of the markers. Here, you can choose to stream labeled markers, unlabeled markers, or both.
  3. [Unity] Make sure the streaming setting is set up correctly, and play the scene.
  4. [Unity] Each marker will be drawn as sphere objects in the scene, as shown in the screenshot below.
Skeleton labeled markers drawn in Unity scene.

Integrating HMDs


OptiTrack motion capture systems can be used to track head mounted displays (HMD) and integrate the tracking data into Unity for unique VR applications. For instructions on integrating HMD tracking data into Unreal Engine, please refer to the corresponding pages:

Info2.png

Supported HMDs

At the time of writing, the following HMDs are supported:

  • HTC Vive
  • HTC Vive Pro
  • Oculus DK2
  • Oculus CV1
  • Oculus GearVR

Live Streaming


Once you have connected the client object and configured the script components on the assets from the scene. Play the scene, and if the properties are correctly configured, the assets in the game scene will be animated according to the position and orientation of the rigid bodies and the HMD in Motive. When playing the scene, the HMD and the assigned actor objects will be placed in respect to their location in Motive’s coordinate system disregards to their position in the editor.

Wireless Multiplayer Setup


When setting up multiplayer games with wireless clients, it is more beneficial for each client to make direct connection to both the tracking-server (Motive) and the game-server, rather than rebroadcasting the streamed tracking data through the game-server. Then, any of the game related actions that interacts with the tracking data can be processed on the game-server, and this server can send out the corresponding updates to the wireless clients. This allows the wireless clients to only receive both the tracking data or updates without having to send back any information; in other words, minimizing the number of data transfers needed. If wireless clients are sending data there will be a minimum of two transfers on the wireless network, and each transfer of data through wireless network is at risk of latency or lost packets.

WirelessClientSetup.png