Replace the Background in Facebook’s Spark AR Studio

Unnecessary background within an image, photograph, or video makes the image lose its focal point. By removing the background, we can draw the attention of the viewer back to the focal point, thereby making the image more informative and easy to view.

In this article, we will learn to create segmentation and the camera texture to create an effect that replaces unnecessary background in Facebook’s Spark AR Studio. We will make use of a segmentation mask texture to separate the foreground and background of the scene. Then, we will implement the camera texture to a material in the foreground to stream a video of the person using the effect back into the scene. You can then transform the environment behind them and replace the background.

The method does not require any custom assets to create this effect. All we need to do is follow these simple steps after opening Spark AR Studio to get started.

Step 1: Create a Segmentation Mask Texture

Get your Spark AR Studio set up by creating the new blank project. We will start by creating the segmentation mask texture. This texture separates the foreground scene with the user/focal point from their background.

To create a segmentation mask texture:

1. Select the option of Camera in the Scene panel.

2. Then go to the Inspector panel and click + next to the Segmentation option.

3. Select the Person option.

You will now notice the texture in the Assets panel, named personsegmentationMaskTexture0.

Step 2: To create the Camera Texture

Now, we will create the camera texture:

1. Ensure you select the Camera in the Scene panel.
2. Then go to the Inspector panel and click + next to Texture Extraction.

You will now see a texture called cameraTexture0 in the Assets panel.

Step 3: Make and Set the Background

To interpret the user in the scene and create the background, we will use two 2D objects called rectangles.

The first rectangle will have a material with the camera texture and segmentation mask texture applied to it, forming the foreground. The other will have a single material applied to it, forming a background.

Rectangles always act as children of a canvas. In this effect, we will make the rectangles rescale with the screen of the device.

We will begin by making the first rectangle that will form the background.

1. Click on the Add Object option at the bottom of the Scene panel.

2. Select a rectangle from the menu. A rectangle will be added to your Scene Panel.

It is a good practice to rename the rectangle according to your requirements. This helps you to keep track of the objects used. To do this:

1. Right-click on the rectangle on the Scene Panel and select the Rename option.

2. Name it background_rectangle.

In the Viewport, the rectangle will appear. But the rectangle is quite small and positioned in the top corner of the scene.

We want the background to fill the whole canvas and therefore the full screen of the device. To do this, go over to the Inspector panel and edit the rectangle’s properties to make the rectangle cover the device screen. Set:

• Width to Fill Width.

• Height to Fill Height.

Next, we need to create a material for the rectangle. To create the material go to the bottom of the Inspector Panel:

1. Click + in front of the Materials option.

2. Next, click the dropdown and select the Create New Material option.

The material will get listed in the Assets panel. Right-click on the material in the Assent panel and rename it. Good practice is to name it “background_material.”

We can achieve many kinds of effects with segmentation, for example, using the custom textures for the background. But, for this article, we will pick a color for the segmentation effect.

For the color segmentation effect, select the material. Go to the Inspector panel and change the Shader Type to Flat.

Then, under the Diffuse option of the Inspector Panel:

1. Click the color box next to the Color option.

2. Choose any color you want for the background from the menu and click OK.

Step 4: Add Segmentation

Next, let us prepare the foreground scene with the user in it.

1. Right-click on the canvas of the Scene panel and add another rectangle.

2. Right-click on the rectangle on the Scene Panel and select Rename option. Name it user_material.

3. Again, go to the Inspector panel and select Fill Width and Fill Height.

4. Now we will create a material for the user_material rectangle and select it in the Assets panel. Like earlier, rename it. You could pick something like user_material.

5. Go to the Inspector panel and change the Shader Type to Flat.

6. Then, under the Diffuse option, click the dropdown next to Texture. Select cameraTexture0 from the dropdown list.

This will make the user appear in the scene, but we can not see the background yet. This is because we have not applied the segmentation mask texture yet.

To apply the segmentation mask texture to your scene:

1. Checkmark the box next to Alpha in the Inspector panel.

2. Next to Texture in the Inspector panel, click the dropdown and select the segmentation mask texture.

We have completed the effect using segmentation to separate the user from the background.

Finally, after following the above steps, you can successfully get a lens with the desired background. Here is a demo lens: