Model Serving

The model is not the slow part

Ask a team where their serving latency goes and most will say the model, because the model is the part they built and the part they profile. Then somebody instruments the whole path and finds the answer somewhere else: the service spent longer fetching the features the model needed than it spent running the model. That is the normal result, not the surprising one, and it changes what you should optimise first.

The practical discipline is to measure the path rather than the call. Put a span around each step — receive, fetch features, preprocess, infer, postprocess, serialise — and look at the percentiles rather than the mean, because a mean hides exactly the requests users complain about. And set a budget in milliseconds before you choose an architecture, since the budget decides whether you can afford a network hop to a feature store at all, and that is a design question rather than a tuning one.

Three shapes of serving, and the cheapest one usually wins

Before building a service, answer a smaller question: does the prediction have to be made while the user waits? A great many do not. If the set of things you predict about is known in advance and changes slowly — customers, products, accounts — you can compute the predictions on a schedule, write them to a table, and let the application read a row. That is not a compromise; it is a serving architecture with no p99, no autoscaling and no cold start.

• Batch (precomputed)

  Score everything nightly, write to a table, read a row at request time. Cheapest to run and to operate. Fails only when the input is not known in advance.

• Online (per request)

  A service that takes input and returns a prediction. Necessary when the input arrives with the request — a search query, a basket, a form. Everything hard on this page is about this shape.

• Streaming

  Predictions attached to events as they flow — a fraud score written onto each transaction. Latency matters but nobody is watching a spinner, which is a much kinder constraint.

A hybrid is common and worth knowing as a pattern: precompute the expensive parts on a schedule and combine them with request-time inputs in a cheap final step. A recommendation system that scores every user against every item nightly and then re-ranks the top few hundred with fresh session context is doing exactly this, and it turns an impossible latency budget into an easy one. Reach for it before reaching for a bigger machine.

Batching, accelerators and the right scaling signal

A GPU is throughput hardware. It reaches its rated performance by doing the same operation across many rows at once, and a single request of one row uses a small fraction of it while occupying the whole device. That is why serving runtimes do dynamic batching: hold arriving requests for a few milliseconds, group whatever showed up, run them as one batch. You trade a small, bounded amount of latency for a large multiple of throughput, and the maximum wait is a number you set rather than a hope.

Two more numbers decide the bill. Model loading time sets how badly a scale-up event hurts: a multi-gigabyte model that takes ninety seconds to load onto a device means autoscaling cannot respond to a spike, so you provision for the peak or you keep warm capacity. And memory, not compute, is usually what limits how many models fit on one device — which is why serving several small models from one runtime, each with its own version, is often the cheapest arrangement anyone finds.

Shipping a version without betting on it

A new model version is not a bug fix; it is a change in behaviour across every request at once, and its offline score is a prediction about how it will behave rather than an observation. So the deployment question is not "is it better" but "how do we find out at a survivable cost". Four mechanisms answer that, and they compose.

• Shadow

  Send real traffic to the new version and throw its answers away, comparing them offline. Zero user risk, and it catches latency, crashes and skew before anyone is affected. Costs double inference.

• Canary

  Route a small share of traffic to the new version and watch the operational metrics. Answers "does it survive production", not "is it better" — that needs an experiment.

• A/B experiment

  A randomised split measured on a business metric with a stated duration and sample size. The only mechanism that answers whether the model is actually better for users.

• Pinned rollback

  The previous version stays loaded and reachable, and the live version is a config value. Rollback then takes seconds and does not require the training pipeline to still work.

The order that works is shadow, then canary, then experiment — each one cheaper to abort than the next. Two details matter more than they look. Serve both versions from one runtime that can hold several versions at once, so switching is a routing decision rather than a deployment. And record the model version on every prediction you log; without it, an investigation three weeks later cannot separate the two populations and the experiment you ran becomes unanalysable.

You cannot watch accuracy, so watch what you can

Here is the uncomfortable fact that shapes model monitoring. Accuracy needs labels, and labels arrive late — a churn label after the subscription period, a fraud label after the chargeback, a recommendation label after the return window. For weeks, the only honest statement about a live model is that you do not know how well it is doing. Building a dashboard around accuracy therefore produces a panel that is either empty or stale, and teams learn to ignore it.

What you can observe immediately is everything except the outcome, and it is more useful than it sounds. Three layers, in the order they catch problems:

• Operational: latency percentiles, error rate, queue depth, saturation. Breaks first and is unambiguous when it does.
• Inputs: the distribution of each feature against the training snapshot, plus the rate of nulls and unseen categories. This is where an upstream change announces itself.
• Outputs: the distribution of predictions and, for classifiers, of the scores. A model whose positive rate doubles overnight is telling you something before any label arrives.
• Business proxies that resolve quickly — click-through, acceptance rate, override rate by human reviewers — as the earliest signal correlated with the outcome you cannot yet measure.

Anti-patterns worth naming

• Predictions logged without the model version, the feature vector and a timestamp — which makes every later question unanswerable.
• An online service built for inputs that were known the night before, paying for p99 that nobody needed.
• Preprocessing reimplemented inside the service rather than shared with the training pipeline — skew by construction.
• A rollback plan that means retraining, since the previous artifact was overwritten and the data has moved on.
• A drift alert with no owner and no defined action, which is disabled within a month of its first seasonal false positive.

When to use it

Related pages

• InfrastructureML PipelinesAn ML pipeline is the automated path from raw data to a model you can deploy. A trained model is a build artifact, not source code. It comes out of four inputs — the data, the training code, the configuration and the environment — and only one of those is in your repository, which is why cloning the repo and running the script gives you a different model. A pipeline exists to make all four recorded, repeatable and inspectable. This page covers the stages and why they are separate, why data breaks silently where code breaks loudly, the skew that makes an excellent offline model useless online, and when a notebook is genuinely enough.
• InfrastructureKubernetesKubernetes is a control loop with an API in front of it. You do not tell it to start a container; you record a desired state, and controllers keep comparing that record to what is actually running and act on the difference — forever, not once. Every object it offers and every surprising behaviour it has comes from that. This page covers the loop, the handful of objects that matter, why resource limits and health probes cause most production incidents, and when you genuinely need any of it.
• DevelopmentLLMOpsLLMOps is the practice of running a feature built on a language model in production, and it exists because everything that makes software releasable — a test that fails, a diff that explains a change, a rollback that restores yesterday — assumes the same input gives the same output. LLM features break that assumption, so the practices have to be rebuilt around it. This page covers what replaces the test suite, what the deployable artefact actually is, what to record on every call, and how cost and latency behave once real traffic arrives.