Quick answer: Face tracking is software that follows the position and movement of facial landmarks, such as the eyes, brows, mouth, and jawline, across video frames in real time. It maps those landmarks to a 3D mesh or a set of expression values so an app can react to them, without trying to work out whose face it is looking at. You meet it whenever a filter puts virtual ears on your head, whenever an Animoji mimics your grin, or whenever a video call app keeps a moving face centered in frame.
This entry covers how face tracking works mechanically, where it shows up in apps you already use, how it differs from facial recognition, and where it tends to break down.
What Is Face Tracking?
Face tracking locates a face in a video frame and then follows specific points on it, frame after frame, as the person moves, talks, or changes expression. Those points might be a simple set of landmarks, like the corners of the eyes and mouth, or a dense mesh with hundreds of points covering the whole face.
The goal is geometry, not identity. A face-tracking system does not ask who this is, it asks where this face’s features are right now and how they are moving. That narrower goal is what lets it run in real time on a phone, using far less processing than a full identification pipeline needs.
Most consumer face tracking also produces expression data alongside the raw geometry, often called blend shapes: numerical scores for movements like a raised eyebrow, an open jaw, or a closed eye. Those scores are what let a digital character mimic your face instead of just wearing a static mask shaped like it.
The models behind this are trained on large sets of images labeled with landmark positions, the same supervised approach used across computer vision generally, so a tracker has effectively learned what a brow or a smiling mouth looks like from many examples before it ever sees your face.
How Does Face Tracking Work?
The pipeline starts by detecting that a face is present and locating it precisely in the frame. After that, the work is continuous: the system predicts where each point has moved, checks that guess against the new frame, and corrects it, dozens of times a second.
Depth-sensing systems get the richest data. Apple’s ARKit, built on the iPhone’s TrueDepth camera introduced with the iPhone X in 2017, projects an infrared dot pattern onto the face and reads how those dots deform to build a coarse 3D mesh matching the face’s real shape and expression. ARKit reduces that mesh to more than fifty blend shape coefficients describing individual movements like a blink or a smile, and that data is what Animoji and Memoji animate against.
Camera-only systems skip the depth sensor entirely. Tools like Google’s MediaPipe Face Landmarker estimate a dense mesh of 478 points across the face from an ordinary 2D video feed using a trained neural network, no infrared projector required, which is why face filters work on laptops and Android phones with no depth camera at all. The tradeoff is that a 2D estimate is inherently a guess about depth, while a TrueDepth-style mesh measures it directly.
Both approaches update dozens of times a second, which is what keeps a filter or avatar feeling attached to your face instead of visibly lagging a beat behind it.
Where Is Face Tracking Used?
Animoji and Memoji are the clearest consumer example: point an iPhone’s front camera at your face and a panda, robot, or custom cartoon avatar mirrors your expressions in real time, entirely through blend shape tracking. Augmented reality filters on Snapchat and Instagram use the same idea, tracking landmarks to warp, decorate, or replace parts of a face as it moves.
Video call software leans on face tracking for background blur and framing, keeping a moving face centered in the crop or a virtual background steady behind it. Beauty and try-on apps use the same landmark data to preview makeup or glasses in the right place on a moving face.
It also underlies more playful AR filter work in games and live streaming, where a tracked face drives an on-screen character or overlay live as a performer talks.
The same idea, following landmarks instead of identifying anyone, extends past the face too. Motion capture applies it to a performer’s whole body for film, game, and sports work, often running alongside face tracking on the same shoot.
Headset-based computing leans on it too. Apple’s Vision Pro scans the wearer’s face during setup and then uses cameras and motion sensors inside the headset to track expressions in real time, driving a digital avatar called a Persona that stands in for the wearer’s face on a video call, since the headset itself covers it.
Face Tracking vs Facial Recognition: What’s the Difference?
This is the pairing people mix up most. Face tracking follows where facial features are and how they move; facial recognition goes a step further and works out whose face it is by comparing it against stored data.
A face filter that gives you cartoon eyes has no idea who you are and no way to find out, since it never compares your face against a database of known identities. The two share their opening step, detecting that a face is present, and only recognition adds an identification stage on top of it.
That is also where their privacy profiles part ways. Tracking generally keeps landmark data on the device and drops it when the camera closes, while recognition exists to match a face against stored identities, which is what puts it under scrutiny that plain tracking mostly escapes.
What Are the Limits of Face Tracking?
Occlusion is the most common failure: a hand passing in front of the mouth, glasses glinting in bright light, or a face turned too far to the side can all cause a tracker to lose landmarks or jitter until the face is fully visible again.
Camera-only systems without a depth sensor are more exposed to this, since they infer 3D shape from a flat image rather than measuring it, so unusual lighting or an extreme angle can distort the estimated mesh in ways a TrueDepth-style system generally would not.
Expression data has its own ceiling. Blend shape scores capture broad movements like a jaw drop or a wink well, and subtler things, a slight asymmetry in a smile or the way skin creases around the eyes, tend to get flattened out, which is why avatar faces can look a little generic even when the tracking is working perfectly.
None of that makes face tracking unreliable for its actual job. It is built to animate an avatar or steer a filter in real time, not to make a high-stakes decision, so a dropped frame costs a moment of visible jitter and nothing more.