ML Pipelines

A model is a build artifact, not source code

Software engineering has a comfortable assumption underneath it: the repository is the truth, and anyone who checks it out and builds gets the same program. Machine learning quietly breaks that assumption. Your repository holds the training code, but the model was produced from that code plus the exact rows of data it saw, plus the hyperparameters someone passed on the command line, plus the versions of the libraries and drivers on the machine that ran it. Change any one of the four and you get a different model — usually slightly different, occasionally very.

This is not a theoretical concern, and it shows up as a specific bad afternoon. A model in production starts behaving oddly. You want to compare it with the previous one, so you look for how the previous one was built — and you find a notebook, a commit from four months ago and a table that has been written to every day since. There is no way to rebuild the old model, so there is no way to tell what changed. That afternoon is what a pipeline prevents, and it is a cheaper reason to build one than any argument about automation.

The stages, and why they are separate

A pipeline is a directed graph of steps where each step reads named inputs and writes named outputs. The separation is not tidiness. It is what lets a step be cached instead of recomputed, what lets a failure name itself, and what lets two runs be compared step by step. Six stages cover almost every training pipeline in production, and each one exists because something goes wrong when it is folded into its neighbour.

• 1 · Ingest

  Pull the raw data and freeze it as an addressable snapshot — a partition, a table version, a manifest of file hashes. "Yesterday’s query" is not an input; a snapshot id is.

• 2 · Validate

  Check the snapshot against an expected schema and expected distributions before spending anything on it. This is the stage teams skip and the stage that would have caught the incident.

• 3 · Features

  Turn rows into the numbers the model consumes. Kept separate because this transformation must run identically at serving time, and separating it is what makes that possible.

• 4 · Train

  The expensive step, and the least interesting one. It reads a config file rather than command-line flags, and it writes an artifact plus the run metadata that names its four inputs.

• 5 · Evaluate

  Score the candidate on a held-out set AND against the model currently in production. A number with nothing to compare it to has never stopped a bad release.

• 6 · Register

  Write the artifact, its scores and its lineage into a registry under a version. Registration is not deployment — it is the hand-off that makes deployment a separate, revertible decision.

Two things about this graph are worth saying plainly. First, the orchestrator is the least important choice on the page — Airflow, Kubeflow, Dagster, Metaflow, Prefect or a Makefile behind your existing CI all express the same graph, and none of them fixes an unaddressable data input. Second, the steps should be ordinary code that runs on your laptop with a small snapshot. A pipeline you can only execute by pushing to a cluster is one nobody will iterate on, and the cost of that shows up as people going back to the notebook.

Code fails loudly, data fails silently

When code breaks, something throws and a build goes red. When data breaks, everything succeeds. An upstream team renames a column and your join produces nulls; a currency field switches from cents to units and a feature silently divides by a hundred; a logging change stops emitting an event and a category disappears from the training set. In every one of those cases the pipeline runs green, the model trains, the metrics move a little, and nobody looks — because nothing failed.

The fix is to make data failures loud, which means writing expectations down and letting the pipeline refuse. The checks worth having are boring and few:

• Schema: the columns exist, with the types and the nullability you expect. Cheap, and catches the rename.
• Volume: the row count is within a range of recent runs. A snapshot that is 3% of yesterday means an upstream job failed, not that demand collapsed.
• Ranges and categories: numeric fields inside plausible bounds, categorical fields drawn from the known set. Catches the unit change and the new enum value.
• Missingness: the share of nulls per column compared with previous runs, since a column that quietly goes 90% null is the classic silent break.
• Distribution drift against the training snapshot, reported rather than enforced at first — a hard gate here fires on Black Friday and gets disabled by March.
• Leakage: no feature computed from information that does not exist at prediction time. The most expensive one to find later and the hardest to automate.

Training/serving skew, and what a feature store is for

Here is the single most common reason a model that scored well offline performs badly online, and it is not a modelling problem at all. Training features are computed in one place, in a batch job, in Python, over a whole table. Serving features are computed in another place, per request, often in another language, over one row. Two implementations of the same idea drift apart — a different rounding rule, a different default for a missing value, a time window that means the last 30 days in one and the last 30 calendar days in the other. The model then sees inputs at serving time that it never saw in training, and it degrades in a way no amount of retraining fixes.

A feature store is the answer to exactly this and to nothing else, which is worth saying because it is sold as much more. Its real job is that a feature is defined once and served two ways: an offline store that can answer "what was this value as of that timestamp" for building training sets, and an online store that answers "what is this value now" in single-digit milliseconds. Buying the product is optional; the discipline is not. A shared library that both sides call, with one test asserting that offline and online produce the same value for the same entity and time, gets you most of the benefit at a fraction of the operational weight.

The registry, and why registering is not deploying

The last stage writes into a model registry, and the useful way to think about it is as an artifact repository that happens to know about models. It stores the binary under a version, together with the metrics it scored, the four inputs that produced it, and a record of which environment currently runs which version. That last field is what makes a rollback a configuration change rather than a retraining job.

Keeping registration separate from deployment is the part teams collapse and then regret. A pipeline that trains and immediately promotes to production has no place to put a human decision, no place to compare the candidate against the incumbent, and no way to answer "put back what we had at nine this morning" except by training again — with different data, since the data has moved on. Separating them costs one config field and buys the two things you always need at the worst moment: a comparison and a way back.

• Version the artifact, never overwrite it. model-latest.pkl in a bucket is not a registry, and it makes every incident unanswerable.
• Store the evaluation next to the artifact, including which dataset it was scored on. A metric without its dataset is not a metric.
• Record the lineage as identifiers, not prose — snapshot id, commit sha, config hash, image digest.
• Make promotion an explicit transition with a name, so "which model is live" is a query rather than an archaeology exercise.

Retraining is a schedule, not an emergency

Models decay because the world moves, not because the weights rot. Prices change, a competitor launches, a new customer segment arrives, and the relationship the model learned six months ago is no longer the relationship in the data. The mature answer is a scheduled retraining run whose result is evaluated against the incumbent and only promoted if it wins — which turns decay from an incident into a routine that mostly produces a "no change" outcome. The immature answer is retraining when someone notices something is wrong, which means the pipeline is exercised rarely and therefore fails when it is needed.

How often is a business question with a technical test. Take the model in production, retrain it on data ending a month ago, and score both on the most recent month. If the fresh model wins clearly, your data moves fast and monthly is a floor; if the two are indistinguishable, you are retraining more often than the world justifies and you are paying for it in compute and in review time. Do this measurement once and it settles an argument that otherwise runs for a year.

Anti-patterns worth naming

• The notebook that became production: cells run in an order nobody recorded, against a table that has since changed.
• Data addressed as "the table" rather than as a snapshot, so the same code produces a different model every day by design.
• A platform built before a second model exists. One model does not need an orchestrator; it needs a script and a pinned environment.
• Feature code duplicated between training and serving, with no test asserting the two agree.
• A candidate promoted because its score beat last week’s, without ever being scored against the model actually running.
• Retraining triggered by a drift alarm alone, so a seasonal week rewrites a model that was fine.

When to use it

Related pages

• InfrastructureModel ServingModel serving is putting a trained model behind an interface that answers requests. Once a model is trained, serving it is a latency and cost problem before it is a machine learning problem. The inference itself is usually the smallest part of the response time, the hardware is throughput hardware being asked to do a latency job, and the thing most teams monitor — accuracy — is the one thing they cannot see live. This page covers the request path, the three shapes of serving and why the cheapest usually wins, batching and the right scaling signal, how to ship a new version without betting on it, and what to watch when the truth arrives weeks late.
• InfrastructureCI/CDContinuous integration is not a server that runs your tests — it is the habit of merging into one shared branch often enough that conflicts stay small, and the automation exists to make that habit safe. Continuous delivery is the second half: one artifact, built once, promoted through environments without being rebuilt. This page covers how to order the stages, why a flaky test destroys the pipeline's authority, what the deployment strategies actually trade, and the four numbers that tell you whether any of it is working.
• DevelopmentFine-tuningFine-tuning continues a model’s training on your own examples, so the behaviour you keep describing in the prompt becomes the behaviour it defaults to. It is excellent at teaching form, tone and judgement calls, and it is close to useless at teaching facts. This page covers where it sits among the cheaper options, what the methods actually differ on, why the dataset is the whole project, and the maintenance bill that starts the day you ship.