Quick answer: Inference latency is the time a trained model takes to respond once it receives a request, measured from the moment input arrives to the moment usable output comes back. It is a performance measurement, not an accuracy one, and it is the main reason one AI app feels instant while another feels sluggish doing the same job. This entry covers what inference latency actually measures, why it means something different for a chatbot than for a camera app, and why running a model on-device changes the number.
What Is Inference Latency?
Every trained model goes through two very different phases. Training happens once, offline, and can take days or weeks. Inference happens every time someone actually uses the model, and inference latency is simply how long that single use takes.
The clock usually starts when a request reaches the model, whether that is a typed prompt, a photo or a frame of video, and stops when the model has produced a response, or enough of one to be useful. That second detail, what counts as “enough,” is where the metric splits into different practical measurements depending on what kind of model you are dealing with.
Latency is also not one fixed property of a model the way its size or architecture is. The same model can show very different latency depending on the hardware running it, how many other requests it is handling at once, and how the input itself is shaped, so a latency figure only really means something alongside the conditions it was measured under.
How Is Inference Latency Measured?
For a model that returns one answer all at once, like an image classifier labeling a photo, latency is a single number: the gap between input and output. For a large language model generating text one token at a time, the picture is more layered.
Two numbers matter more than the overall total. Time to first token is how long a user waits before anything appears on screen at all, and it dominates how fast a chatbot feels to talk to, since a response that starts appearing quickly reads as responsive even if the full answer takes several more seconds to finish. Tokens per second describes how quickly the rest of the response streams in once it has started, and its inverse, inter-token latency, is the same measurement stated as a per-token delay.
Vision and audio models add their own version of the same idea. A camera app processing live video is measured in frames per second, and it needs to keep up with the rate the camera captures at, commonly 24 to 30 frames a second, or the feed visibly stutters. Falling behind that rate is immediately obvious to a user in a way that a slow text response is not.
Why Does Inference Latency Matter for Chatbots and Vision Apps?
A chatbot like ChatGPT streams its answer onto the screen token by token rather than making a user stare at a blank box until the whole reply is ready. That design choice exists specifically because time to first token, not total response time, is what a person actually notices while waiting.
A vision app lives by a different clock. A live camera filter, a real-time object detector in a retail app, or a driver-assistance system checking the road ahead all need a fresh prediction roughly every frame, which means inference has to finish in a few dozen milliseconds rather than a second or two. Miss that window consistently and the feature stops looking real-time and starts looking broken, even if each individual prediction is accurate.
Face ID is a small, everyday example of the same pressure. The comparison against stored face data has to finish fast enough that unlocking a phone feels instant, which is part of why that kind of matching runs on dedicated hardware like the Apple Neural Engine rather than general-purpose silicon.
Inference Latency vs Throughput: What’s the Difference?
Latency and throughput answer different questions, and it is easy to optimize one while quietly hurting the other. Latency asks how long one single request takes. Throughput asks how many requests a system can handle in total over a stretch of time.
A server can often raise throughput by batching many requests together and processing them as a group, which uses the hardware more efficiently overall but can make any one request in that batch wait slightly longer than it would running alone. A chat product tuned for a single user typing in real time usually favors low latency, while a backend processing millions of requests overnight usually favors throughput instead, and the two goals genuinely pull in different directions.
A team building a live product usually has to pick which one they are optimizing for before they start tuning hardware or model size, since the settings that help one metric routinely cost something on the other. Reporting only one of the two numbers is also a common way vendor benchmarks end up misleading, since a system can post an impressive throughput figure while individual users still wait noticeably long for their own response.
On-Device vs Cloud: What Changes Inference Latency?
Running inference in the cloud means the request has to travel over a network to a server, get processed, and travel back, and that round trip adds time on top of whatever the model itself needs to compute. Running inference on the device removes that round trip entirely, since there is no network hop between the input and the answer.
That is a large part of why on-device AI and dedicated chips like an NPU matter as much as they do. Local hardware is usually slower than a data-center GPU on raw compute, but skipping the network trip can still leave it faster end to end for a small model, especially on a shaky connection where the round trip itself is the bigger cost.
The tradeoff runs the other way for large models. A phone or laptop cannot fit the biggest cloud-hosted models in memory at all, so an app chasing top-tier answer quality over raw speed often still calls out to a server despite the extra network latency, accepting the round trip because no on-device model is capable enough to skip it.
What Are the Limits of Chasing Lower Latency?
Latency numbers reported by a vendor are usually measured under close to ideal conditions, so real-world latency on a busy server or a weak connection is typically worse than the headline figure suggests. A model can also be sped up in ways that quietly trade away quality, since a smaller or more aggressively compressed model generally answers faster but less accurately than the full version it was built from.
There is no single acceptable number either, since the right target depends entirely on the task in front of the model. A chatbot has more slack than a driver-assistance system checking the road, and treating every use case as though it needs the same speed usually means overspending on hardware somewhere it does not matter.