Data platforms
A data platform is the stack that collects, stores and moves an organisation's records, and one built for a reliable network sends first and stores as a side effect. Invert that assumption and the architecture inverts with it: the node writes locally and always, sending becomes an opportunistic background activity that may not happen today, and the interesting design decision is which records leave first when the contact window is forty seconds wide. This page covers that queue, the receiving side that expects to be interrupted, and the three properties every record has to carry.
- Write
- Locally, always
- Send
- By priority, when possible
- Every record carries
- Provenance and a class
Store and forward, and the queue that makes it a design
Store and forward is old and simple: write it down where you are, send it when you can. What makes it a design rather than a buffer is what happens when the queue is bigger than the window — which is the normal case, not the exception.
Do the arithmetic once and the point becomes obvious. A forty-second window at nine kilobits per second moves about forty-five kilobytes. The critical event is two kilobytes and goes every time. The track updates are forty kilobytes and go most times. The video is nine hundred megabytes and will never go over this link at all — which is a design fact to be decided in advance, not a surprise to discover in the field. Video leaves on physical media when the platform returns, or it is summarised on board into something that fits, or it does not leave.
The receiving side assumes it will be interrupted
Because the window closes without warning, every transfer is a transfer that might be half finished, and the connection that resumes may be a different one, from a different node, out of order. Three properties make that survivable, and all three are cheap to build in and painful to retrofit.
| Property | What it means concretely | What breaks without it |
|---|---|---|
| Idempotent ingest | Every record has a stable id generated at the edge; receiving it twice is indistinguishable from receiving it once. | A retry after an interrupted window produces duplicates, and the count of everything is quietly wrong. |
| Order independence | Records carry their own event time and are placed by it, not by the sequence they arrived in. | Yesterday’s batch arriving after today’s rewrites the present with the past. |
| Append, never overwrite | New information is a new record referencing the old one; nothing already stored is replaced. | A late correction deletes the original observation, and the disagreement that mattered is gone. |
| Resumable transfer | Progress is tracked per record, so the next window continues rather than restarting. | A queue larger than a window never drains, because every attempt starts from the beginning. |
Together these make the sync protocol an exchange of immutable facts rather than a reconciliation of mutable state, which is a considerably simpler thing to get right. It is also the property the C2 layer depends on: correlation can only be re-run over the union of what both sides saw if both sides kept what they saw as records rather than as a current value.
Three things that must travel with every record
Each of these looks like metadata and each one determines something structural. Adding any of them later is a migration across every producer, every store and every consumer, which is why they belong in the first schema.
Provenance
Which device, which sensor, which software version, which operator, when. A record without a source cannot be judged, cannot be corrected and cannot be excluded when its source turns out to have been faulty.
Classification
The handling category, which decides which storage the record may legally sit in, who may read it and where it may cross a border. It is an architectural input, not a label added at export time.
A retention rule
How long it is kept and what happens then. Here deletion is an obligation somebody will audit rather than a way to reduce a storage bill, and it has to be enforceable on a device you may not reach.
Classification is the one that surprises engineers, because it moves a legal question into the architecture. If a category of record may not leave a jurisdiction or may not sit on a shared service, then that constraint decides where the store lives, how the sync is routed and which managed products are available to you — and it decides it before any performance consideration does. Getting the categories written down early is what stops a working system from being unshippable for reasons that have nothing to do with whether it works.
Anti-patterns worth naming
- A first-in-first-out queue, so the oldest routine telemetry leaves ahead of the alert raised thirty seconds ago.
- Record ids generated on the receiving side, which turns the first interrupted transfer into permanent duplicates.
- Overwriting on correction, which deletes the original observation and with it the evidence that anything disagreed.
- No expiry on records, so a queue accumulates position updates from two hours ago that will never be worth sending.
- Treating classification as an export-time label rather than as the input that decides where the data may live.
When to use it
Use it when
- Whenever the network is an event rather than a utility and a node must keep working through a long absence.
- When the queue is larger than the contact window, which makes the priority order a product decision.
- When designing ingest — idempotent, order-independent, append-only and resumable, all four together.
- When writing the schema, since provenance, classification and retention are migrations if added later.
Avoid it when
- As general data engineering material — the databases and infrastructure sections own that and this page leans on them.
- As guidance on the shared picture built from this data, which is the C2 page rather than this one.
- Where the network is genuinely reliable, in which case store-and-forward is complexity you are not being paid for.
- As a reason to keep everything forever, since retention here is an obligation rather than a storage decision.
Found this useful?
Share it with someone who is working on the same problem.