Add a Hand Tracker and 3D Object in Facebook’s Spark AR Studio

Gesture recognition helps us to know the meaning of human body movement, including the movement of hands, head, arms, face, or body. Today’s world is observing huge advances in visualization displays and this includes object trackers in AR applications.

The main purpose of this article is to explore gesture recognition in Facebook’s Spark AR Studio for an efficient human-computer interaction. It presents a method for tracking hand gestures and recognizing hand gestures by extracting unique invariant features from gestures.

We can use a hand tracker to generate effects that respond when the camera detects a hand. In this article, we will learn to add hand tracking to an effect and tracking the object in hand.

In the finished effect we will notice that when we wave our hand, it will look like we are holding a baseball.

Step 1: Set up your new project in Spark AR Studio

To do this:

1. Open the Spark AR Studio and click on ‘New Project.’

2. A new tab containing different types of projects will appear. Select ‘Blank Project.’

Step 2: Setting up the interface

To view the effect with hand tracking, we will need to switch on our computer’s camera. This will let us see our own hands in the Simulator when we test out the hand tracking effect.

To switch on your camera in Spark AR Studio:

1. Click the video symbol in the toolbar present in the extreme left of Spark AR Studio’s interface.

2. Then, select your camera.

Step 3: Adding a hand tracker to the scene

To start adding a hand tracker to the scene:

1. Click the Add Object option at the bottom of the Scene panel.

2. Then, choose the Hand Tracker option from the list.

3. This will add a Hand Tracker to our Scene panel. In the Simulator, we will see an instruction saying “Hold up a hand!”

Note: This is an automated instruction that’s added to all effects with a hand tracker. If you hold up a hand towards the camera, the message will disappear.

Before moving to step 4, let us take a look at some of the properties of the hand tracker.

When we have selected the hand tracker in the Scene panel, we will see its properties in the Inspector Panel on the right of the screen.

We can adjust these properties to make changes to our Hand Tracker.

When we see a small dropdown arrow next to a property, we can click it to create a patch representing the property in the Patch Editor.

• Layer: This property assigns the Hand Tracker to a layer, or creates a new layer.
• Visible: Through this property, we can stop the Hand Tracker from being rendered in the scene.
• Transformations: We cannot change the position, scale, or rotation of a Hand Tracker, as these sub-properties are controlled by the position of the face detected by the face tracker.
• Interactions: This property helps us to insert simple combinations of patches into the Patch Editor.
• Enable For: This property lets us choose the camera or cameras on a mobile device that we want to render in the Hand Tracker.

Step 4: Tracking an object to a hand

To make an object detect the hand, we will need to make the object a child of the hand tracker in the Scene panel. To do this, we can do either of the two things:

• Go to the Hand Tracker in the Scene panel, right-click and select desired 3D Object.
• Drag an object from the Assets panel to the Hand Tracker in the Scene panel.

We have added a 3D object to the project. It’s listed in the Assets panel as HireSphere.

You can also try out hand tracking with other amazing 3D objects from the Spark AR Library.

To make the object appear in response to a hand movement, the object should be a child division of the Hand Tracker in the Scene panel. When an object is a child division of another object in Spark AR Studio, it responds to its parent object — in this case, appearing when a hand is detected.

To create a child-parent relationship between the Hand Tracker and the 3D object, all we need to do is drag the 3D object onto the hand tracker option in the Scene panel.

Now, as the 3D object is a child of the Hand Tracker, it is nestled underneath the Hand Tracker in the Scene panel.

And there we have it, an effect with hand tracking.