Hand-Eye Coordination

The Hand-Eye Coordination example showcases two gaze interactions:

  1. Gaze assisted throwing
  2. Picking up objects with gaze

This example also shows a slightly advanced usage of Tobii G2OM, which will be explained in more detail later on this page.

Interaction in this example is done by using the right VIVE controller, which is the first controller that was turned on. The Trigger button is used to pick up and throw objects, and the Menu button is used to reset the scene.

Otherwise, open up the Hand-Eye Coordination scene found in TobiiXR > Examples > HandEyeCoordination_Example > Scenes.

To learn about how to design for the Hand-Eye Coordination use case or eye tracking in general, check out our design section.

Compatibility

This particular Unity Example is compatible with the following headsets:

HMD Compatible Instructions
Tobii HTC VIVE Devkit Yes Getting Started
HTC VIVE Pro Eye Yes, follow instructions Getting Started
Pico Neo 2 Eye No

Table of Contents


Scene Contents

The root objects in the scene are:

  • Tobii XR Manager Hand-Eye has the TobiiXRThrowingSceneManager component attached which initiates the SDK and the Tobii G2OM.
  • Tobii XR Player Hand-Eye has the Unity scene camera, and the ControllerManager component which handles the controller movements and inputs. The GazeGrab and ThrowAtGaze components are also attached, which handle gaze assisted grabbing of objects and throwing of objects respectively.
  • Environment contains the objects in the scene which are non-interactable, such as the room and lighting.
  • Gaze Throw Targets contains objects which act as potential targets in the gaze assisted throwing.
  • Grabbable Ball is a ball which the user can pick up by looking at it and pressing the trigger.
  • Throwable Ball can be picked up by looking at it and pressing the trigger, and will also assist the user when throwing it at one of the Gaze Throw Targets.

The blue ball is the Throwable Ball, the gray ball is the Grabbable Ball and the blue rectangles in the scene are the Gaze Throw Targets.


Tobii XR Manager

In this scene, a custom G2OM instance is created in the TobiiXRThrowingSceneManager which is located on the Tobii XR Manager Hand-Eye game object. This custom G2OM instance allows for dynamically switching which type of objects should be gaze focusable. There are two types of objects which are interactable with gaze:

  • The grabbable balls, which can be picked up by looking at them and pressing a button.
  • The gaze throw targets, which can be thrown at when a ball is grabbed to the hand.

The G2OM instance is instructed to only map gaze to the two grabbable balls if no ball is being held. If a ball is being held, the G2OM instance is instead instructed to map only to the targets. Doing this separation allows for better object selection, and objects which are not interactable can be filtered out before being sent into G2OM. The TobiiXRThrowingSceneManager has a reference to the GazeGrab component on the Tobii XR Player Hand-Eye game object to be able to tell if the user is currently grabbing a ball.


Visual Feedback

The GazeOutline component is placed on the Grabbable Ball and the Throwable Ball to give an outline as visual feedback when the user focuses at the object. This is to let the user know that it is interactable, which in this case means that it can be picked up by pressing the Trigger button.


Grabbing with Gaze

In the scene, there are two objects, Grabbable Ball and Throwable Ball which can be grabbed by looking at the object and pressing the Trigger button on the VIVE controller. The logic for grabbing in the scene is handled by the GazeGrab component which is located on the Tobii XR Player Hand-Eye game object. The GazeGrab component looks for objects of the type GazeGrabbableObject to see if any are focused when the trigger is pressed.

The GazeGrab uses TobiiXR.FocusedObjects to see if any GazeGrabbableObject is being focused. When the user focuses on a GazeGrabbableObject, the GazeGrab component stores the object for up to 100 ms, unless the user focuses on another object of the GazeGrabbableObject type. Adding this extra time allows for the target to lose focus when the user presses the trigger to grab it, which could be because the user has looked away just before pressing the trigger. The extra time added also allows gives more room for the G2OM instance to miss the object at the moment when the user presses the trigger.

The GazeGrabbableObject component has two values which are modifiable in the inspector; the animation time for when the object flies to the hand when grabbed, and also the animation curve of the fly animation.


Throwing with Gaze

In the scene is also an example of gaze assisted throwing. This example only works on non moving targets.

For throwing with gaze to work, three components are required:

  • GazeThrowableObject, which is attached to the object the user should be able to throw using gaze assist.
  • GazeThrowableTarget, which is attached to objects that the user should be able to hit with the thrown object.
  • ThrowAtGaze, which will adjust the throw trajectory within the bounds set by the GazeThrowableObject to hit the GazeThrowableTarget.

The ThrowAtGaze will take the information of the thrown object and the target and call a native plugin to do calculations for adjusting the throw. The ThrowAtGaze has the same behavior as GrabAtGaze when selecting which object which should be focused, meaning the focus lingers on the last object for up to 200 ms. When throwing, this extra linger time can help to keep focus on the correct target at the moment when the user releases the object from the hand.

The Throwable Ball has the GazeThrowableObject component attached to it, which exposes a few values in the inspector which can be changed to modify its behavior when thrown. The values exposed in the inspector define how much assistance ThrowAtGaze gives when throwing. Modifying these values allows for changing the behavior of the gaze assist from always hitting no matter what, to hitting only when it is very close but would not hit without the adjustment.

  • Lower Velocity Multiplier is the lower multiplier from the original to the adjusted throw velocity. A value of 0.5 means that the script will adjust the velocity down to half the original throw velocity.
  • Upper Velocity Multiplier is the upper multiplier from the original to the adjusted throw velocity. A value of 2 means that the script will adjust the velocity up to twice the original throw velocity.
  • Max Y Velocity Modifier is how much the Y component (the up direction) of the throw will be adjusted from the original throw. A value of 1 will adjust the y component value of maximum one unit (which will equate to 1 m/s using standard Unity scale).
  • Xz Angle Threshold Degrees is how many degrees left or right of the target the original throw is allowed to be.

Create Your Own Scene

To add gaze grabbing or throwing to a new or existing scene, follow these steps:

The prefabs are located in TobiiXR > Examples > HandEyeCoordination_Example > Prefabs.

  1. Drag in the Tobii XR Player Hand-Eye prefab, containing the camera and necessary components for throwing, grabbing, controller input and visualization.
  2. Drag in the Tobii XR Manager Hand-Eye prefab to set up the SDK and Tobii G2OM with correct parameters.
  3. Drag the GazeGrab script, located on the Tobii XR Player Hand-Eye, as the Gaze Grab Component for the the TobiiXRThrowingSceneManager on the Tobii XR Manager Hand-Eye game object.
  4. Drag in one or several Gaze Throw Target game objects and place them in the scene. If you want to use other meshes, you could also attach the GazeThrowTarget component.
  5. Drag in the Throwable Ball or the Grabbable Ball prefab.