Building a Custom Glasses Snapchat Lens with Fritz AI and Lens Studio — Zero Code Involved

Computer Vision — SnapML

Introduction

A few weeks ago, Snapchat released a new feature where AI developers can easily integrate their own custom models — one of these features is the ability to segment regions of the view and project fun and immersive AR effects. We’ll cover this ML task in this article.

Lens Studio, an augmented reality creativity suite released by Snapchat in 2017, gives users the opportunity to create their own augmented reality (AR) Lenses.

This software is built for designers or even developers and artists who wish to work with a powerful AR creation tool, with the aim of providing the Snapchat community with ever more immersive and fun lenses. Lenses are a big part of the user’s experience inside Snapchat, used either in stories or in a peer-to-peer fashion, whether to communicate with friends or to simply have fun.

In this article, I will show you how easily you can create a custom machine learning-powered lens with NO code. Then, we’ll create a lens that will segment in real-time glasses and give you the option to change their color. We will use Fritz AI to generate a dataset, annotate, and train a custom segmentation model.

Overview:

1. Fritz AI Studio
2. Lens Studio
3. Evaluation of the results
4. Conclusion

Fritz AI Studio

First, let’s work through building a Lens Studio-ready model with Fritz AI Studio.

• Seed images: Download images containing a portrait of people with glasses, preferably with different shapes and images with different genders and skin colors. I managed to find around 55 good seed images of portrait images with great quality on Unsplash. We’ll use these “seed” images to generate a trainable dataset snapshot.

• Remove the background: Plenty of services offer a way to remove image backgrounds for free, such as remove.bg or clippingmagic.com. If you don’t want to use these services, Photoshop can be used, as well as Preview (macOS only) or GIMP. Fritz AI studio will use these images to overly them on thousands of random images. I have used remove.bg because it’s simple and fast. In my experience, it’s better to have a full image with some context as you will get a chance to annotate part of the image — the glasses.

• Annotate: When all the images are ready, go to Datasets -> Add Image Collection -> Upload images. When all images are uploaded, a whole new menu at the top will appear with an image annotation interface. The process is a bit tedious and time-consuming (as data labeling tends to be), especially for segmentation models. You start by creating new classes, with each one having a different color. In my case, I have only one class (glasses) — it took me around 10 minutes using the keyboard shortcuts to annotate 55 images.

• Generate a snapshot: This is where all the magic happens. With your labeled seed images, Fritz AI will create synthetic images based on your original images, with some sort of data augmentation built-in. You can also monitor how many images have been generated. You will receive an email when the process is finished. I chose at first to create a relatively large dataset because I know from experience that segmentation models will take time to converge and start learning.

I have also tweaked the size of the foreground images because usually in my experience with Snapchat, the segmented subject will be closer to the camera, and thus the scale of the training images should resemble the real use case. This parameter will translate to the seed images covering between 60% to 100% of the final dataset images generated. Additionally, by default Fritz AI will generate around 20% of images with no seed images in order to prevent overfitting of the final model and allow it to generalize.

• Train a segmentation model: When the snapshot is ready, you can start the training job by selecting your snapshot and choosing the number of hours for the training budget — note that Fritz AI will send you an email when the training process is finished. It will also stop the training if the model converges before the assigned training hours. In this case, the model took 5 hours and 27 minutes.

Lens Studio

Now that we have our ML model, we need to import it into Lens Studio and create our lens.

• Open the project: Open the .lsproj file from the project zip file provided by Fritz AI Studio. A prompt will pop up — just click on import.

• Import the model as an ML Component: In the left Objects panel of Lens Studio, you will find an object called ML Component. Click on it and import the model from the right panel. Since the model file is already in the project structure, Lens Studio is able to recognize it.

• Change the input Texture: In the ML Component file, change the input texture to Textures > Device Camera Texture.

• Generate an output Texture: In the ML Component file, click on the Create Output Texture button at the bottom of the right panel to generate an output Texture.

• Final step: Select the output texture as the segmentation model output, it will actually filter out regions of the images based on the segmentation prediction.

Evaluation of the results

The segmentation model works as expected, predicting regions where the glasses are. The drawback is related to the fact that the model will predict the region even though there are no glasses in the view. This is due to the fact that we don’t discriminate faces without any glasses.

Conclusion

While the model is good, the whole workflow could be improved by adding another layer with a model that could classify whether the person has glasses or not. When the classification model detects a face with glasses, then the segmentation model kicks in. In addition, I would also continue to add more seed images to improve the accuracy of the model and generate more and more good quality annotated images.

The project is nowhere near ready to be used by end-users, there are a lot of things that could be added to improve the whole experience in terms of design, or even other elements like music. The possibilities are endless and are likely easy to implement for Snapchat Lens Creators. There is also room to maybe substitute the color effect with images or gifs!

Thank you for reading this article. If you have any questions, don’t hesitate to send me an email at [email protected].