Model Serving

Model serving is the practice of taking a trained machine learning model and making it available in production so that live applications can send it data and get predictions back. Training produces a set of learned parameters; serving is what turns those parameters into a running service that answers real requests — a recommendation for a shopper, a fraud score for a transaction, a caption for an image. In practice, serving means wrapping the model behind an endpoint, usually a REST or gRPC interface, loading it into memory on a server or cluster, and handling each incoming request within a latency budget the application can tolerate. Tools such as TensorFlow Serving, TorchServe, NVIDIA Triton, and KServe exist for exactly this job. Model serving is the operational bridge between a model that works in a notebook and one that does useful work for real users.

Model serving matters because a model that never leaves the data scientist's laptop creates no value. The gap between a trained model and a served one is where most machine learning projects stall. Serving forces a set of hard questions that training ignores: How fast must a prediction come back? How many requests per second must the service absorb? What happens when traffic spikes, or when the model file is many gigabytes? How do you roll out a new version without breaking the old one? Getting these right is the core of MLOps — the discipline, short for machine learning operations, that treats models as software to be deployed, monitored, and maintained. Good serving keeps latency low, scales with demand, and lets teams swap models safely, so the intelligence built in training actually reaches the product.

The clearest way to place model serving is against model training, its upstream sibling. Training is the offline, compute-heavy phase where an algorithm learns patterns from historical data, adjusting parameters until the model performs well on examples it has seen. It can run for hours or days on large clusters, it tolerates delay, and it happens relatively rarely. Serving is the opposite in almost every respect: it is online, it must answer in milliseconds, it runs continuously, and it is judged on latency, throughput, and uptime rather than on accuracy against a test set. A model is trained once, or retrained on a schedule, but served millions of times. Confusing the two leads to bad engineering — sizing serving infrastructure like a training job, or expecting a training pipeline to meet a live request under a tight budget.

Serving is sometimes called inference, and the word is worth keeping straight. Inference is the act of running a forward pass through the trained model to produce an output; serving is the whole system that makes inference available reliably at scale — the endpoint, the load balancer, the autoscaling, the versioning, the monitoring. You can perform inference in a script with no serving layer at all, but you cannot serve a model in production without inference happening inside it. Serving also splits into patterns: online serving answers requests one at a time in real time, while batch serving scores a large set of records on a schedule when latency does not matter. A fraud check needs online serving; a nightly propensity score for an email list can use batch. Choosing the pattern is part of serving the model well.

Serving a model well starts with the latency and throughput the application actually needs, then works backward to the infrastructure. A model behind a checkout page has a tighter budget than one feeding a weekly report. From there the craft is in the operational details: keeping the model warm in memory so the first request is not slow, batching requests where it helps throughput, using hardware acceleration for heavy models, and caching predictions that repeat. Versioning is central — you want to deploy a new model beside the old one, shift traffic gradually, and roll back instantly if quality drops, which is why canary and shadow deployments are common. Monitoring closes the loop: track latency, error rates, and the distribution of inputs and outputs, because a model can keep serving while silently drifting as the world it learned from changes.

The failures cluster around treating serving as an afterthought. Teams train a strong model and then discover it is too slow or too large to serve within budget, or they ship it with no versioning and cannot roll back a bad release. They forget monitoring, so model drift — the slow decay of accuracy as live data diverges from training data — goes unnoticed until the predictions are visibly wrong. They over-provision expensive accelerators for a model that a smaller instance would serve, or under-provision and fall over under load. The discipline is to design serving as a first-class system from the start: define the latency and scale targets, choose online or batch to match, version and monitor every deployment, and retrain on a cadence, so the model keeps earning its place rather than quietly rotting behind a live endpoint.

Model serving names the software practice of exposing a trained machine learning model as a running service for inference, a core concern of MLOps.

Etymology: source.

Search interest for this term over the last five years:

View interest-over-time on Google Trends →

Areas of marketing where model serving is a core concern: