Computer vision
Computer vision is the practice of turning camera images into decisions a machine can act on, and the model is rarely the hard part of it. What makes vision hard here is that it runs on a device with a power budget and no network, on images that do not look like the ones it was trained on, and that its output feeds a decision where the two kinds of error are paid for by different people. This page covers the edge budget, the data problem that no amount of training fixes, the threshold as a policy decision, and what has to be recorded so a wrong answer can be investigated at all.
- Runs
- On the device, alone
- Hardest input
- The one you never trained on
- The threshold is
- A policy, not a metric
Inference happens inside a budget, and the budget is not GPU memory
Model serving in a data centre optimises for throughput per dollar. On a device the constraints are different in kind, not in degree, and three of them have nothing to do with computation. The general serving material lives in the MLOps pages; what follows is only what changes.
| Budget | What it actually limits |
|---|---|
| Power, in watts | Every watt spent on inference is flight time or run time not spent. The model competes with the mission, not with other tenants. |
| Heat | A sealed enclosure in the sun throttles. A benchmark on a bench is not the same number as a benchmark in the field after twenty minutes. |
| Latency, per frame | If the platform is moving, a detection that arrives 400 ms late describes a place the object has left. Latency is an accuracy problem here, not a comfort one. |
| Memory and storage | Decides whether you can keep the raw frames that a later investigation will need, which is usually the first thing sacrificed and the first thing missed. |
| The update path | A model you cannot replace for six months has to be right for six months, which changes how much you are willing to specialise it. |
The data problem, which no amount of training fixes
A detector learns the distribution it was shown. Deployed, it meets a different one, and the gap is not random noise — it is systematic, it is large, and it is predictable enough to be planned for. This is the single biggest source of field failures in vision systems and it is almost never a modelling mistake.
- Sensor shift. A model trained on colour imagery does not transfer to thermal, and one trained on one thermal camera often does not transfer to another. The sensor is part of the training data, not a detail of it.
- Season and weather. Foliage, snow, mud, rain on the lens, low sun straight into the sensor. A model validated in summer has been validated on a quarter of the year.
- Geometry. Altitude, angle and focal length change the number of pixels on an object by an order of magnitude, and a detector trained at one scale quietly fails at another.
- Deliberate concealment. Unlike almost every other domain, someone is actively working to make detection fail — with cover, with decoys, with disruptive patterns. The distribution is not merely different; it is adversarially chosen.
- Class imbalance in the extreme. The thing you care about may appear in one frame in fifty thousand, so a model that always says "nothing here" scores 99.998% and is worthless.
The practical response is not a better architecture. It is a data pipeline that closes the loop: every deployment records real frames, a review process labels the interesting ones — especially the misses and the false alarms — and those enter a permanent evaluation set organised by condition. That is the same discipline as the regression suite described in the QA section, applied to a model, and it is what turns field experience into something the next version inherits rather than something an engineer remembers.
The threshold is a policy decision with a named owner
A detector does not output "yes" or "no". It outputs a number, and somebody turns that number into a yes or a no by choosing where the line goes. That choice is usually made by whoever was tuning the model, on a Tuesday, using the validation set — and it is the single most consequential decision in the system.
Both failure directions have consequences that leave the model behind. Too many false positives and the operator stops trusting the alerts — not through negligence but through arithmetic, because a stream that is mostly noise cannot be attended to. That failure looks like a human problem and is caused by a number. Too many misses and nothing at all happens: no alert, no record, no feedback, and the system quietly reports excellent precision while the thing it exists to find goes past. Of the two, the second is worse precisely because it produces no signal that anything is wrong.
One more property belongs in that policy and is usually absent: the system needs a way to express "I do not know". A detector forced to answer with a binary on every frame will answer confidently on the frames it has never seen anything like, which is the exact opposite of what you want. An explicit uncertain band — below the alert threshold, above the ignore threshold, routed to review rather than to an operator — costs little and is where the interesting failures show up first.
What has to be recorded for a wrong answer to be investigable
A detection that cannot be reconstructed cannot be reviewed, and a system whose mistakes cannot be reviewed does not improve — it accumulates opinions about itself. Recording is constrained here by storage and by the fact that the device may not come back, so it has to be designed rather than switched on.
The frame, or a crop of it
Without the image there is no investigation, only argument. If storage forbids keeping everything, keep the detections and a sample of the misses rather than a uniform sample of nothing in particular.
The model identity
Which model version, which weights, which threshold, which preprocessing. Six months later this is what distinguishes a model regression from a changed environment.
The conditions
Sensor, altitude, angle, time, temperature. These are what let you slice performance by condition afterwards, which is the only useful way to read it.
What the human did
Confirmed, rejected, ignored. This is free labelled data of exactly the right distribution, and most systems throw it away.
The last card is the one worth acting on this week. Operator decisions are a labelled stream produced by the exact deployment you care about, at no annotation cost, and they are the shortest path from "the system is wrong sometimes" to a case in an evaluation set. Capturing them requires the interface to make confirm and reject equally easy, which is a design decision made in the C2 layer rather than in the model.
Anti-patterns worth naming
- Reporting one accuracy number averaged across conditions, which hides the cases the system was built for.
- Benchmarking on a workstation and shipping to a sealed enclosure in the sun, where the same model runs at a third of the rate.
- A threshold chosen by whoever tuned the model, with no written record of what was traded away.
- No uncertain band, so the detector answers confidently on inputs it has never seen anything like.
- Discarding operator confirmations and rejections, which is free labelled data of exactly the right distribution.
When to use it
Use it when
- When inference must run on the device, inside a power, heat and latency budget measured end to end.
- When the deployment conditions differ from the training set by sensor, season, geometry or deliberate concealment.
- When setting a detection threshold, which needs a written policy, both error rates and a named owner.
- When designing what a device records, so a wrong answer six months later can still be investigated.
Avoid it when
- As general model training or serving guidance — the AI/LLM and MLOps pages own that and this one leans on them.
- As a basis for letting a model decide rather than detect — the decision boundary belongs to the overview and to C2.
- With a single aggregate accuracy figure, which is the metric most likely to be both impressive and misleading.
- Where a person can review every frame anyway, in which case most of this machinery is overhead.
Found this useful?
Share it with someone who is working on the same problem.