Quick answer: Model deployment is the step where a trained machine learning model moves out of a notebook or training environment and into a place where it can actually make predictions for real users or systems. That can mean an API endpoint a mobile app calls in real time, a batch job that scores millions of records overnight, or a model packaged directly into an app running on someone’s phone. A model that never gets deployed, no matter how accurate it tested, has not actually done anything useful yet.
Deployment sounds like a formality but it is often the hardest part of a project. This entry covers the main ways teams deploy models, where each approach shows up in practice, how deployment differs from the broader discipline of MLOps, and where it tends to go wrong.
What Is Model Deployment?
Training produces a model: a file containing learned parameters that can turn an input into a prediction. Deployment is everything required to make that file usable outside the environment it was trained in, packaging it, giving it somewhere to run, and connecting it to whatever needs its predictions.
That packaging usually means wrapping the model in a container, most often with Docker, so it runs the same way regardless of the machine underneath it, and orchestrating those containers with a tool like Kubernetes so the service can scale up under load and recover if an instance fails. The specifics vary a lot depending on where and how fast the predictions are needed.
This is where a lot of projects stall, and the model is rarely the reason. A team that has already proved the model works on held-out data can still spend months on packaging, infrastructure, and wiring it into the systems that are supposed to call it.
How Does Model Deployment Work?
The core choice is real-time versus batch. Real-time, or online, deployment stands up a persistent endpoint that accepts a request and returns a prediction immediately, which is what powers a fraud check at checkout or a recommendation shown while someone browses. It demands infrastructure that can respond in milliseconds and stay up continuously.
Batch deployment instead runs the model against a large stored dataset on a schedule. That suits work like scoring an entire customer list overnight, where nobody needs the answer instantly and processing everything at once is cheaper.
A third pattern, edge or on-device deployment, skips a server call entirely and runs the model directly on the device making the request, a phone, a browser, a piece of factory equipment. That removes network latency and keeps data local, at the cost of running within whatever compute and memory the device actually has, which usually means the model has to be shrunk first through quantization or a similar compression step.
Dedicated serving software handles the real-time case at scale. TensorFlow Serving and TorchServe wrap a model in a standard API, and NVIDIA’s Triton Inference Server does the same across TensorFlow, PyTorch, ONNX, and others at once, adding request batching and multi-model hosting so one server can carry several models efficiently.
Cloud platforms like Amazon SageMaker bundle these pieces together, offering real-time endpoints, serverless inference that scales to zero when idle, and shadow testing that runs a new model against live traffic alongside the current one before it takes over.
Replacing a live model safely is its own small problem. Rather than switching every request over to a new version at once, teams commonly roll it out gradually, sending it a small slice of live traffic and comparing its results against the model it is replacing before trusting it with everything, or running it silently alongside the current model without acting on its predictions at all. Either approach catches a model that looked fine in testing but behaves badly on real traffic before it can do much damage.
Where Is Model Deployment Used?
On mobile and other consumer hardware, Apple’s Core ML is a concrete example of on-device deployment: it lets a model trained elsewhere run directly on an iPhone, iPad, or Mac, using the device’s CPU, GPU, and Neural Engine, with no network round trip and no user data leaving the device.
Converting a trained model into a lighter local format like that is what on-device AI amounts to at deployment time: the same learned weights, shipped somewhere else to run.
Server-side, e-commerce recommendation engines and fraud detection systems are classic real-time deployments, since a delayed answer is close to a useless one. Large batch deployments show up in places like credit scoring or churn prediction, where a company scores its entire customer base on a schedule rather than one prediction at a time.
For anyone evaluating AI tools rather than building the platform behind them, deployment is usually invisible until it is not: a chat app that answers instantly is very likely calling a real-time endpoint on a server somewhere, while a photo app that tags faces the moment a picture is taken, with no network delay and no upload, is almost certainly running the model on-device. The gap between those two experiences is entirely a deployment decision, not a difference in how the underlying model was trained.
Model Deployment vs MLOps: What’s the Difference?
Deployment is a single step; MLOps is the whole system around it. MLOps covers the pipeline that trains, tests, versions, deploys, and monitors a model on an ongoing basis, and it treats deployment as one stage in that larger loop, not the end goal. A team can deploy a model by hand once without any MLOps practice at all; MLOps is what you build once that has to happen repeatedly and reliably.
Deployment is also frequently confused with model serving specifically, though the two overlap rather than match exactly. Serving usually refers to the runtime layer that answers prediction requests once a model is live, tools like TensorFlow Serving or Triton. Deployment is the broader act of getting the model into that serving layer in the first place, including packaging, infrastructure setup, and rollout.
What Are the Limits of Model Deployment?
Getting a model deployed does not mean it stays accurate. The data it sees in production can drift away from what it was trained on, and nothing about the deployment itself catches that; it takes separate monitoring to notice a model has quietly gotten worse.
Infrastructure cost and complexity are real constraints too. A real-time endpoint that has to stay available around the clock costs money whether or not it is handling traffic, and teams often over-provision to be safe against demand spikes. Edge deployment trades that cost for a harder engineering problem: fitting a useful model into the memory and compute budget of a phone or embedded device without gutting its accuracy.
Integration is usually the quiet, unglamorous obstacle. A model that performs well in testing still has to be wired into existing systems, authentication, logging, the actual application calling it, and that plumbing work is frequently what determines whether a technically sound model ever reaches a real user at all.