Quick answer: World tracking is the feature in a phone or headset’s augmented reality software that figures out where the device is and how it is moving through a real space, so a virtual object stays locked to the same spot on your table or floor as you walk around it. It is what keeps a digital couch from sliding around the moment you tilt your phone. On iOS this is Apple’s ARKit world tracking; on Android it is Google’s ARCore motion tracking, and both are built on the same underlying idea.
Alongside that position estimate, the system builds a rough sense of the surfaces around the device, which is what lets an app drop an object onto a real floor instead of into thin air. This entry covers how that works mechanically, where you run into it, how it relates to the broader SLAM technique underneath it, and where it tends to fall apart.
What Is World Tracking?
World tracking is the part of a mobile augmented reality platform responsible for one specific job: keeping the relationship between the virtual scene and the real one consistent as the device moves. Everything else in an AR app, the 3D model, the lighting, the tap gestures, sits on top of that foundation.
Apple introduced ARKit with world tracking at its 2017 developer conference, shipping it with iOS 11. The technology traces back to Apple’s 2015 acquisition of Metaio, a Munich-based augmented reality company whose computer vision and SLAM work became part of ARKit’s foundation. Google’s answer, ARCore, does the equivalent job on Android and explicitly describes its motion tracking as an application of SLAM, simultaneous localization and mapping, to a single moving phone camera.
How Does World Tracking Work?
The core technique is often called visual-inertial tracking. The system watches the camera feed for visually distinct feature points, corners, edges, textured patches, and follows how they shift from frame to frame, while reading the phone’s accelerometer and gyroscope to sense its own motion.
Combining the two beats either source alone. The camera can lose its grip during fast motion or blur while the motion sensors stay responsive, and the motion sensors would drift badly over a few seconds without the camera correcting them.
ARKit expresses the result as six degrees of freedom, three rotation axes for roll, pitch, and yaw, and three translation axes for movement in each direction, updated continuously as you move the device. ARCore does the same job under a different name, aligning a virtual camera’s pose with the real one so that content overlaid on the scene appears to hold still in physical space.
Both platforms build on that raw pose estimate with environmental understanding. Clusters of feature points on a flat surface get exposed as detected planes, horizontal ones for tables and floors, vertical ones for walls, so an app can snap an object to a real surface instead of leaving it floating. Apple represents each detected plane as an anchor object; Google calls the underlying surfaces and feature points trackables.
On devices with a LiDAR scanner, ARKit goes further and builds an actual 3D mesh of the room through scene reconstruction rather than just flat planes. ARCore adds light estimation, reading the average brightness and color of the scene so a virtual object can be shaded to match its surroundings instead of looking pasted on.
Anchors matter because the underlying pose estimate keeps refining itself as the system sees more of the room. Without an anchor tying a virtual object to a specific point in the world, that object could visibly drift as the platform’s estimate of the room improves. Anchoring locks it in place relative to that improving map instead of to a single, possibly wrong, early estimate.
Where Is World Tracking Used?
World tracking is the foundation under nearly every mobile AR experience. Furniture and home goods apps use it to let you see how a couch or a lamp would look in your actual living room before buying. Games use it to place characters and effects that appear to interact with your real floor and walls.
Retail and virtual try-on apps use it to keep a product registered on a surface, or on your face and hands, as you move the phone around it.
The platform side has grown alongside these apps. ARKit runs on iPhones and iPads with an Apple A9 chip or newer, which covers essentially every device Apple has sold since world tracking launched, and Apple extended the same technology to visionOS when it shipped its Vision Pro headset in 2024.
ARCore covers a wide range of Android phones certified to run it. Between the two, world tracking is the layer nearly every consumer-facing augmented reality app quietly depends on, whether the app developer thinks about it directly or not.
World Tracking vs SLAM: What’s the Difference?
SLAM is the general technique: building a map of an unknown space and locating yourself in it at the same time, using whatever sensors a given machine has, a robot vacuum, a delivery drone, a self-driving car. World tracking is what happens when that same idea gets specialized for a single handheld or head-worn camera plus a phone-grade inertial sensor, wrapped in a developer-facing API, and tuned for the specific job of keeping a virtual object visually locked to a real one.
Put another way, world tracking is SLAM’s most common consumer-facing form, but it is a narrower job than SLAM in general. A robot’s SLAM stack usually needs to build and keep a durable, precise map of an entire building for navigation. A phone’s world tracking session typically only needs a good enough local sense of nearby surfaces and its own short-term motion, since most AR sessions last minutes, not the lifetime of a warehouse robot’s map.
What Are the Limits of World Tracking?
Feature-point tracking needs something to grab onto, so blank, uniformly colored surfaces such as a plain white wall can be difficult for the system to register reliably. The same goes for dim or flat lighting, where the camera struggles to pick out distinct points to follow from frame to frame.
Fast or erratic motion causes trouble too. A sudden whip of the phone can blur the camera image badly enough that the visual half of the tracking briefly loses its footing, leaving the system to lean on motion sensor data alone until the camera can reacquire enough features to recover.
None of that makes world tracking fragile in ordinary use. It has become reliable enough that most people using an AR shopping app or game never think about the sensor fusion running underneath, which is really the point: the technology is doing its job precisely when nobody notices it is there.