When Snapchat’s Lens Studio released SnapML as a core part of their 3rd major platform update (3.0) back in June of 2020, our antennae immediately raised at the introduction of a new mobile platform for machine learning models.
We seized the opportunity, got to work, and less than a year later, a collection of our machine learning models have been baked into Lens Studio 3.4, the latest release of Snapchat’s flagship design tool. That’s right—Lens Creators can now directly access ML models designed and built specifically for Lens Studio by our team of experts!
You can read much more about the full Lens Studio release in our complete coverage, linked below, but here, we wanted to highlight the Fritz AI models that made the cut and are accessible through Lens Studio’s new “Asset Library”.
About Lens Studio’s Asset Library
First, a quick note about the new Asset Library in Lens Studio.
For folks who want to get started with a range of different capabilities inside Lens Studio, but don’t want to go elsewhere to build 3D objects, ML models, materials, and more, this new library is filled with a bunch of ready-made assets that you can quickly add to your projects—including 2D and 3D assets, scripts, materials, audio files, and of course, a variety of ready-to-use SnapML models.
Fritz AI Models in the Asset Library
At launch, there are 6 Fritz AI models included in the Asset Library.
Of course, using Fritz AI’s no-code model building Studio, you can also generate new datasets and train your own custom models for each of the ML tasks included here (Image Labeling/Classification, Object Detection, Image Segmentation), as well as unique Style Transfer effects—all without any ML overhead.
But the SnapML models included in the new Asset Library are a great entry point for learning what different ML tasks do and how they can be used in your Lenses. Here, I’d like to briefly introduce you to the Fritz AI models included in the first version of the Asset Library:
Dessert Classification
If you have room for dessert, you can use this template to help users indulge with style.
The presence of an ice cream cone could trigger the siren song of an ice cream truck, or a pie could spawn a rainfall of 3D apples.
Mug Detection
My favorite time of day is early morning — that first cup of tea or coffee. Using this Object Detection template, you could attach 3D objects directly to your cup of Joe, or have other AR effects interact with the mugs.
Pet Segmentation
Ok, so solid green and blue cats and dogs might not be the thing to stick up on Mom’s fridge, but with this template, you can isolate cats and dogs, and identify their boundaries in the scene, and add AR effects or experiences directly to them.
Pine Tree Segmentation
While we originally built this one for the holiday season, it’s also true that pine trees are part of the evergreen family for a reason. Because this model creates a mask containing all the tree’s pixels, you can ornament all parts of any tree exactly the way you want—no matter the season.
Face Mask Segmentation
Face masks aren’t going away anytime soon, unfortunately — and for good reason. Wear a mask!
With this template, we can at least make it fun, though…you can change the color with a slider as seen below, or experiment with other AR effects to transform face masks.
Pet Face Detection
It’s true, we love our furry friends at Fritz AI (try saying that quickly 5 times 🤪). That’s why we also build an object detection model to locate and track the faces of your pets (cats and dogs only).
This model allows you to center your focus on just your pets’ furry faces—like Junebug here!
What’s Next?
With the new Asset Library in Lens Studio 3.4, custom ML from Fritz AI is now baked directly into Lens Studio. Creators have easier and more convenient access to powerful, unique machine learning features than ever before.
And once you get a taste for what ML can do in your Lenses, there’s still plenty more to do. With free beta access to Fritz AI, you’ll find another slew of pre-trained SnapML projects, a Style Transfer model training tool, and the ability to train completely custom models of your own—all without code or ML overhead.
Comments 0 Responses