Logging
Logging is the record a running system leaves behind so a question about it can be answered later — and it is the part of operations most often done in a way that cannot be used. The test is not how much you record — it is whether a question asked three weeks later has an answer, and that is decided when the code is written rather than when the incident starts. This page covers why a log line should be an event rather than a sentence, what logs are for compared with metrics and traces, the fields that make an incident answerable, and the arithmetic that turns a logging habit into a bill nobody predicted.
- A log line is
- An event, not prose
- Decided when
- The code is written
- Cost driver
- Volume × retention
A log line is an event, not a sentence
Most logging code is written for an imagined reader: a person scrolling a terminal at the moment the thing goes wrong. That reader almost never exists. The real reader is a query, typed weeks later by someone who was not there, asking how many users this affected, whether it started with the last release, and whether it is still happening. A sentence cannot answer any of those. A record with fields can answer all three.
The practical change is small: emit an object rather than a formatted string, and put every fact you interpolated into the message into its own field as well. Failed to charge card for user 8213 after 3 attempts becomes an event named payment_failed with user_id, attempts and reason beside it. Keep the human-readable message too if you like — nothing is lost — but the fields are what make the record countable. And use one consistent field name across services, because user_id in one service and userId in another is two columns to a query engine and one very annoying afternoon to a human.
Logs, metrics and traces answer different questions
Teams that log everything usually do so because logs are the only signal they have, and they are paying for it in storage while still not being able to answer basic questions. The three signals are not three formats of the same thing; they have different shapes, different costs and different limits, and knowing which question belongs to which is most of the skill.
| Signal | The question it answers | What it costs, and where it stops |
|---|---|---|
| Metrics | "Is something wrong, and since when?" Numbers aggregated over time — rates, latencies, counts. | Cheap and constant per series, so keep them for a long time. They cannot tell you about one specific request. |
| Logs | "What exactly happened to this one thing?" The individual events, with their details intact. | Cost scales with traffic, so retention is a budget decision. Poor for trends, since counting them is expensive. |
| Traces | "Where did the time go, across services?" One request as a tree of timed spans. | Usually sampled, because keeping every trace is not affordable. Weak for anything rare unless sampling is smart. |
The useful discipline follows from the table. Alert on metrics, because they are cheap enough to evaluate continuously and stable enough not to fire on one bad request. Diagnose with traces, because they show where the time went without you guessing which service to look at. Confirm with logs, because that is where the detail of the individual failing case lives. A team that alerts on log searches is paying a query engine to do a metric’s job, and it will be both slower and more expensive at exactly the moment it matters.
The fields that make an incident answerable
A short, boring list covers most of it. Every event, from every service, should carry these — not because a standard says so, but because each one is a question somebody will ask while a customer is waiting.
Trace / correlation id
The single most valuable field on this page. It ties every log line from every service back to one user action. Generate it at the edge, propagate it everywhere, log it always.
Event name
A stable, low-cardinality name from a closed set. This is what you count, group and alert on when you eventually turn a log into a metric.
Service and release
Which code produced this. Without the release, "did the deploy cause it" is unanswerable — and that is the first question in most incidents.
Actor and tenant
Who it happened to, as an id rather than a name or an email. Needed to answer scope: one customer, one region, or everyone.
Outcome and reason
Succeeded or failed, and a reason code from a closed set. "Failed" alone forces a human to read the message; a reason code can be grouped.
Duration
How long the operation took, in milliseconds, as a number. It costs nothing to record and answers "was it slow before it broke" for free.
Two things should never be in a log. Personal data — names, emails, addresses, card numbers, tokens, full request bodies — because logs are copied to a search platform, kept for months, and read by more people than your database is, which makes them the most likely place for a leak that no one audited. And anything you cannot afford to have printed, which in practice means credentials: a log line is the classic path by which a secret escapes a system that was otherwise careful. Log an id, then look the record up in the database when you actually need it.
The arithmetic that becomes a bill
Logging cost is one multiplication, and doing it once prevents the surprise. Take requests per second, times log lines per request, times bytes per line, times seconds in a day. A service at 500 requests per second writing 8 lines of 600 bytes each produces about 200 GB a day — from one service. Multiply by the number of services, then by the retention period, then by whatever the platform charges for ingestion and for indexing, which are usually billed separately. Most teams meet this number for the first time in an invoice.
Four levers control it, and they are worth pulling in this order:
- Stop logging what nobody queries. Most volume is a handful of lines in a hot path, added during a debugging session years ago and never removed.
- Sample the successful and keep all the failures. Nobody needs every one of a million identical successes, and everybody needs every failure.
- Tier retention: searchable for a fortnight, cheap archive for the compliance period. Almost every query is about the last few days.
- Turn the highest-volume events into metrics and stop storing them individually — a counter answers "how many" for a rounding error of the cost.
Making one incident answerable end to end
In a system of one service, logs are a file. In a system of several, they are a pile of files that describe the same user action from five directions and cannot be joined — unless something was carried between them. That something is a trace id, generated at the first point the request enters your system and passed onwards in a header, then attached to every log line the request produces anywhere. It is a couple of hours of plumbing and it is the difference between reconstructing an incident and speculating about it.
Two further habits make the difference between a log store and a usable one. Emit the same field names everywhere — agree them once, put them in a shared library, and do not let each service invent its own. And use one open convention rather than a vendor’s format: OpenTelemetry gives you a trace id, a span id and a set of standard attribute names, which means the collection and the platform can be replaced without touching application code. That second point matters more over time than it looks now, because the log platform is the component teams most often want to change and least often can.
Anti-patterns worth naming
- Messages assembled from templates, so every line is a unique string and nothing can be counted without a regular expression.
- Logging the exception and then rethrowing it, so one failure appears four times at four levels of the stack.
- Personal data in fields, kept for ninety days on a platform with a broader access list than the database it came from.
- DEBUG left on in production after an investigation, discovered the following month by the finance team.
- No trace id, so an incident across services is reconstructed by comparing timestamps and hoping the clocks agree.
- Alerting on a log search, which is a metric’s job done by a query engine at the worst possible moment.
When to use it
Use it when
- Anything running unattended, since the log is the only account of what happened that survives the process.
- Investigating an individual case — this order, this user, this request — where an aggregate cannot help.
- Recording business events that need to be counted or audited later, where the event name is the unit of analysis.
- Any system where more than one service touches a request, provided a trace id is carried between them.
Avoid it when
- As a source for alerts or dashboards that a counter would serve better, faster and for a fraction of the cost.
- As a store for anything personal or secret, because retention and access are both wider than you think.
- On a per-item hot path, where the volume costs more than the information is worth and slows the request itself.
- As a substitute for tracing when the question is where the time went — a log has no notion of a span.
Found this useful?
Share it with someone who is working on the same problem.