Data platforms

Twelve ways a disconnected platform loses data, and the design for each

Here is the whole list first, because a disconnected platform rarely fails in the lab and usually fails in one of these twelve ways in its first months in the field — quietly, with every dashboard green. Each row pairs what you will see with what actually happened and the design that prevents it. The rows are numbered so a review can say "we have nothing for row 9", and the sections below refer to them by number.

Store and forward, and the queue that makes it a design

Store and forward is old and simple: write it down where you are, send it when you can. What makes it a design rather than a buffer is what happens when the queue is bigger than the window — which is the normal case, not the exception.

Do the arithmetic once and the point becomes obvious. A forty-second window at nine kilobits per second moves about forty-five kilobytes. The critical event is two kilobytes and goes every time. The track updates are forty kilobytes and go most times. The video is nine hundred megabytes and will never go over this link at all — which is a design fact to be decided in advance, not a surprise to discover in the field. Video leaves on physical media when the platform returns, or it is summarised on board into something that fits, or it does not leave.

That decision is a table somebody has to write, class by class. The sizes below are illustrative; the columns are not. The column people leave out is expiry, and it is what row 3 is about: a position update is only worth sending until a newer one exists, so state should supersede itself in the queue rather than wait its turn behind itself.

The same logic applies to the node’s own storage, which is a queue too. A node offline for a week fills its disk, and with no budget the first thing to go is whatever arrives next — often the alert. Give each class a storage budget: routine data is evicted oldest first, critical data never, and every eviction is counted, so "we dropped 4,000 position updates" is a line in a report rather than a discovery.

The receiving side assumes it will be interrupted

Because the window closes without warning, every transfer is a transfer that might be half finished, and the connection that resumes may be a different one, from a different node, out of order. The design question is therefore not the first window but the second: what happens to the records that were in flight when the first one closed. That is rows 2 and 4, and three common answers are drawn below.

Four properties make interruption survivable, and all four are cheap to build in and painful to retrofit.

Together these make the sync protocol an exchange of immutable facts rather than a reconciliation of mutable state, which is a considerably simpler thing to get right. It is also the property the C2C2Command and control — command and control softwareSoftware that turns reports from many sources into one shared picture people can act on, and carries the decisions made from it. Its central job is correlation — deciding whether two reports describe one thing or two — which is why it is harder than the map it is usually drawn as.Full article →All terms layer depends on — its checklist line 3.1, an append-onlyappend-onlyrecords added, never changedA store or log in which new records are only ever added and existing ones are never changed or deleted in place; a correction is a new record pointing at the old one. It costs space and turns "the current value" into a query, and in return nothing that was once known can silently disappear.All terms log of reports rather than a diff of tracks. Correlation can only be re-run over the union of what both sides saw if both sides kept what they saw as records rather than as a current value.

What travels with every record

Each of these looks like metadata and each one decides something structural. Adding any of them later is a migration across every producer, every store and every consumer, which is why they belong in the first schema. Read the last column as the reason: each field is there because a row of the opening table happens without it.

Classification is the field that surprises engineers, because it moves a legal question into the architecture. If a category of record may not leave a jurisdiction or may not sit on a shared service, that constraint decides where the store lives, how the sync is routed and which managed products are available to you — and it decides it before any performance consideration does. Nor is it a property of single records only. In January 2018 a fitness app’s public heatmap, built from a billion harmless runs, turned out to trace soldiers’ routes around bases whose layout was not public. A platform that classifies record by record also needs a rule for aggregates, and the export that builds one is where to enforce it.

Retention cuts both ways, which is row 9. Deletion here is an obligation somebody will audit, not a way to reduce a storage bill, and it has to reach the copy on a device you may not see again — so the rule travels with the record and the device enforces it on its own. Backups and archives are the part usually forgotten; encrypting each retention period under its own key and destroying the key when the period ends is how a deletion reaches copies nobody can list. The other half is just as real: in January 2021 a defective weekly deletion job on the UK Police National Computer removed hundreds of thousands of records it should have kept. A deletion run is previewed, logged and reversible for a grace period before it becomes final.

Evidence is a deliverable: what the system knew, and when

In this field, showing that something works is a deliverable with its own effort estimate: test records, traceability from each requirement to the test that proves it, an argument for why the safety case holds. The data platform carries the same obligation after the fact. When something goes wrong, the question is never what is true now. It is what the system knew at a given moment, what it showed the person who acted, and whether anyone has changed that record since.

The mechanism is two times on every record, which is row 11. Event time is when something happened; recorded time is when the platform learned it. Keep both, never update in place, and "as known at 14:02" becomes an ordinary query rather than an impossible one. SQL:2011 standardised both kinds of time in tables, and several databases implement it; an append-only log with two timestamps gets you the same property without depending on any one of them.

Twenty years of formats

Ten to twenty years in service is normal here, and it changes every data-format decision. The records a system writes in its first year have to be readable in its last, by software that has not been written yet, maintained by people who have never seen the original program. The diagram puts numbers on that: by the vendors’ own policies, everything that reads the data will be replaced several times over while the data stays where it is.

The cautionary tale is from 1986. The BBC marked the 900th anniversary of the Domesday Book with a digital survey of Britain, stored on laserdiscs that only one particular computer setup could read. By 2002 working hardware was so rare that the content had to be rescued by emulation. The Domesday Book itself, written in 1086, was readable the whole time.

• Open, documented formats with a published specification — not the internal file of one product, and not whatever the current library happens to serialise.
• The schema travels with the data or is referenced from it by version, so a file explains itself without the program that wrote it.
• Schemas evolve by rules an older reader tolerates: fields are added, never repurposed, and an unknown field is preserved rather than dropped.
• A sample of every schema version is archived and read back by the new reader on every release — row 12, as a test.
• Storage media are migrated on a schedule, while drives for them are still easy to buy, not when the last one fails.

Seven cases, and the row each one teaches

Defence data failures are rarely published, so these cases are civilian, for the same reason as on the computer vision page; the mechanisms transfer exactly. Two of the seven are about data that survived rather than data that was lost, because that is the other half of the lesson: what investigators had was what had been written locally, or was small enough to leave first.

Where it degrades

The honest limit is that store and forward only decides the order of loss. A node destroyed before its window loses whatever it held; the priority queue decides only which records had already left. Nothing in the platform recovers an observation that was never recorded, and no queue makes a nine-kilobit link carry video. The honest design says in writing what is lost first when things go wrong, rather than implying that nothing is.

• Priority classes where every producer marks its own records critical, which quietly turns the queue back into first in, first out.
• An "offline mode" tested by unplugging a cable for five minutes.
• Provenance fields that exist in the schema and say "unknown" in most rows.
• Everything classified at the highest level, so nothing can be shared and people copy data out by hand.
• Append-only in the database, and an export job that overwrites.
• A retention policy that exists as a document and is enforced nowhere, least of all on devices.
• An archive nobody has read back since the day it was written.
• A sync dashboard that shows success rates but not counts, so a transfer that dropped half its rows reports success.
• "Keep everything forever" presented as caution, when it is a retention decision nobody made.

When to use it

Related pages

• DeftechC2 systemsA command and control (C2) system is software that turns reports from many sources into one shared picture people can act on, and it is usually pictured as a map. It is better understood as a correlation engine with a map attached, whose central job is deciding whether two reports are one thing or two. The page opens with the twenty-five-line shared-picture checklist, then covers correlation, the two times and the clock every record carries, what happens when two nodes lose contact, interoperability as a requirement, the person in the loop as properties you can test, and seven documented failures.
• DeftechCybersecurityCybersecurity for a defence system is the set of design decisions that keep it trustworthy against a funded, patient adversary who may hold the device itself. Ordinary application security assumes an opportunist on the far side of a network boundary and a patch that ships on Tuesday; here the boundary eventually fails completely, and the patch may take months to arrive. The page opens with the twenty-five-line survivability checklist, then covers the threat model and what it accepts, the captured device, the boot chain and the update path, the supply chain, twenty years in service, eight documented compromises, and where the practice turns into theatre.
• DeftechDrone softwareDrone software is the flight code on an uncrewed aircraft and the ground software commanding it, built to stay safe when the link, position or battery fails. An uncrewed aircraft is a distributed system in which one node falls out of the sky if it misses a deadline, and that one fact decides which code runs where, what may block and what may crash. The page opens with the twenty-five-line flight-readiness checklist, then covers the stack as a hierarchy of deadlines, the failsafe ladder as the real specification, the return decision as an energy calculation, autonomy as a list of decisions, simulation as the test environment, and seven documented accidents.