Microservices

What the network adds — sixteen problems, the pattern that answers each, and what it costs

Inside one process, a call is a function call. It takes nanoseconds, and it either returns or throws. Data lives in one database, so one transaction keeps two tables consistent. Splitting a system into services turns every one of those calls into a network call, and every shared table into somebody else’s data. Each change creates a problem the monolith never had. Every one of those problems has a known pattern, and every pattern has a price. Here are sixteen of them, with the point at which you need each. Later sections cite the rows by number.

What makes a service a microservice — nine characteristics, and the one that defines it

Ignore the “micro” for a moment. It has caused more damage than any other syllable in this field, because it invites people to measure services in lines of code. The property that matters is a yes-or-no question. Can this service be changed, tested and deployed to production without deploying anything else? If yes, it is a microservice. If a release needs two services to go out together, you have one service in two repositories. Sam Newman puts it in one line: microservices are independently deployable services modelled around a business domain.

The term was defined on 25 March 2014, when James Lewis and Martin Fowler described “an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms”. They did not give a formal definition. Instead they listed nine characteristics that the systems they had seen tended to share. Most explanations skip the list, and that is a pity, because each item names a way the style is broken in practice. Here are all nine, in their order:

The data rule is older than the word. In 2006 Amazon’s CTO Werner Vogels described the service-oriented architecture the company had moved to after 2001, when its monolithic front end could no longer scale. Each service encapsulated its data with the business logic that operates on it. “No direct database access is allowed from outside the service,” he said, “and there’s no data sharing among the services.” In the same interview he gave the ownership rule its slogan: “You build it, you run it.” Lewis and Fowler cited the ownership slogan eight years later, and their sixth characteristic restates the data rule. Little about the rules has changed since; what has changed is how much tooling exists to follow them.

What you need before the second service

On 28 August 2014 Martin Fowler published a short list of what a team needs before it builds microservices, and announced it as “You must be this tall to use microservices” — the sign at a fairground ride. The list has four items. None of them is exotic, and a single application can live without any of them for years. Twenty services cannot live without them for a week. Here they are, with a test you can run on your own team today:

Nine months later Fowler named the cost behind the list: the microservice premium. Automated deployment, monitoring, failure handling and eventual consistencyeventual consistencycopies agree — laterA guarantee that copies of the data will agree once writes stop arriving, with no promise about when. Until then a read can return an older value than the one just written, so a user may not see their own change on the next screen unless that read goes to where the write was made.Full article →All terms are work that a monolith does not need, and the work starts before any benefit arrives. So a simple system is less productive as microservices, and only past a certain level of complexity does the style pay for itself. His conclusion was blunt: “don’t even consider microservices unless you have a system that’s too complex to manage as a monolith.” The Monolith page shows what to try before you reach that point, and the Monolith → Microservices page shows how to move once you are past it.

Data across services: no joins, no shared transactions

The rule is simple: each service owns its store, and no other service touches it (row 7). The consequences are where most of the real work lives. They surprise people who arrived expecting the hard part to be deployment. A monolith gave you several guarantees for free, and each of them now has to be rebuilt — or consciously given up. The table lists what you lose and what usually replaces it.

Sagas: a transaction you write yourself

Distributed transactions exist. Two-phase commit has a coordinator ask every participant to prepare and then to commit, and databases have supported it for decades. Between services it is rarely used, for three reasons. Every participant, including the message broker, must support it, and many do not. Participants hold their locks while they wait. And if the coordinator fails at the wrong moment, they stay blocked until it comes back. So the usual answer is older and simpler. Hector Garcia-Molina and Kenneth Salem described it in 1987 for long-running transactions inside one database, and called it a saga.

A saga is a sequence of local transactions. Each one commits in its own service, then triggers the next. Every step that can be undone has a compensating step that undoes it — not a rollback, but a new action with the opposite business meaning. An order saga creates the order as pending, reserves stock, charges the card, then confirms. If the charge is declined, the saga runs backwards: it releases the stock and marks the order rejected. Chris Richardson’s “Microservices Patterns” gives the order of steps a useful vocabulary. Compensatable steps come first. Then comes the pivot — the step after which the saga only moves forward. Every step after the pivot must be retried until it succeeds.

There are two ways to coordinate the steps. In orchestration, one component — a service or a workflow engine — tells each participant what to do next and knows the whole flow. In choreography, nobody knows the whole flow: each service listens for events and reacts. The choice decides where the saga’s state can be seen and where its bugs will hide.

The outbox: saving and publishing as one step

Every saga step ends with two writes: the step’s own data, and a message telling the next service to go. That pair is where data quietly goes missing (row 9). Save the order, then publish “order created” — and if the process dies between the two, the order exists and nobody else ever hears of it. Publish first, then save — and a crash leaves an event about an order that does not exist. Two writes to two systems are not a transaction, however close together they sit in the code.

The transactional outbox removes the gap. The service writes the event into an outbox table in its own database, in the same transaction as the order itself. Either both rows commit or neither does. A separate relay then reads the outbox and publishes each event. It can poll the table, or it can read the database’s own change log — change data capturechange data capturestreaming changes from a database logStreaming every insert, update and delete from a database’s own change log to other systems, as tools such as Debezium do. It keeps a second copy in sync without the application writing twice, at the price of coupling every consumer to the source’s table layout.All terms, the approach tools such as Debezium take. The relay may crash after publishing and before marking the event sent, so it will sometimes publish twice. That is why row 10 exists: every consumer records the ids it has processed and ignores a repeat.

All of this inherits eventual consistency as a product question, not a technical one. If stock is reserved a second before the payment clears, someone has to decide what the customer sees during that second, and what happens if the payment then fails. That decision belongs to the business. It has to be made explicitly for every flow that crosses services, and pretending it away is how systems end up charging people twice.

Calls between services: timeouts, retries and the availability bill

Every call between services can be slow, can fail, or — the worst of the three — can succeed on the other side while returning an error to you. A synchronous call also couples availability. Suppose a request passes through a gateway, then orders, then billing, then a ledger. If each is up 99.9% of the time — about 43 minutes of downtime a month — the chain is up 99.6%, which is nearly three hours a month. That is worse than any component in it, and worse than the monolith that did the same work in one process.

An asynchronous edge breaks that coupling. If checkout publishes an event and search consumes it, search can be down for an hour and no customer notices at checkout; search catches up when it returns. The cost is that search shows stale results for that hour, and someone has to be comfortable with that. Here is the trade in one sentence: synchronous calls buy freshness and pay in availability, asynchronous ones buy availability and pay in staleness.

Start with the timeout (row 1), because without one nothing else works: a caller with no timeout holds a thread until the dependency answers, and a dead dependency never does. Amazon’s practice, as Marc Brooker describes it, is to choose an acceptable rate of false timeouts — say 0.1% — and set the timeout at the matching latency percentile of the callee — the 99.9th percentile in that case. Pass the remaining time downstream as a deadline, too. Otherwise a service five hops down keeps working for a caller that gave up long ago.

Retries (row 2) are where good intentions turn into outages. A retry helps with a failure that will not repeat, such as a restarted instance. Against an overloaded service it adds load exactly when load is the problem. Retries also multiply. Brooker’s example is a request that passes through a five-deep stack of services, with three tries at each layer. If the database at the bottom starts failing under load, it receives 3 × 3 × 3 × 3 × 3 = 243 queries for every one the customer made. Amazon’s rule is to retry at a single point in the stack, with exponential backoffbackoffwaiting longer between retriesWaiting before retrying a failed call, and waiting longer after each failure — for example 1, 2, 4 and 8 seconds — usually with a random jitter added so that many clients do not retry at the same moment. Without it, retries arrive exactly when the struggling service can least afford them.All terms and jitter — waits that grow and are randomised, so that clients do not retry in step.

A cap on retries keeps a bad hour from becoming a bad day. Google’s SRE book describes two. Each request gets at most three attempts. And each client keeps track of what share of its requests are retries, and stops retrying once that share reaches 10%. In Google’s worst-case example, the first rule alone still lets traffic grow to almost three times normal; the two together hold the growth to about 1.1 times. A per-client token bucket does the same job, and AWS has built one into its SDKsSDKSoftware Development Kit — ready-made tools for a platformA packaged set of libraries, tools and examples for working with someone else’s platform. The difference from an API is that an API only states the rules, while an SDK hands you code that already knows how to speak them.All terms since 2016.

Circuit breakers (row 3) are the best-known answer and the most argued about. Michael Nygard named the circuit breaker and the bulkhead in “Release It!” in 2007. A breaker counts failures, and once they cross a threshold it stops calling the dependency for a while and fails fast instead. Netflix’s Hystrix library, open-sourced in 2012, made both patterns standard in the Java world. On 19 November 2018 Netflix put Hystrix into maintenance mode at version 1.5.18. Its focus had moved to adaptive concurrency limits, which react to measured latency instead of thresholds set by hand, and it pointed new projects at resilience4j. Amazon goes further in its own guidance. Breakers add a second mode of behaviour that is hard to test, Brooker writes, and they can add significant time to recovery. Amazon prefers to limit retries with a local token bucket instead.

The last row is the one juniors are never warned about and every senior has been burned by (row 5). A network failure does not tell you whether the operation happened: the call may have succeeded and only the response was lost. So any retry of anything that changes state must be safe to repeat, which means the receiver has to recognise a request it has already processed. The client sends a unique idempotency keyidempotency keya request id that makes a retry safeA unique value a client sends with a request that changes something — a payment, an order — so the server can recognise a repeat and return the first result instead of acting twice. The server has to store the keys and decide how long to keep them; a retry that arrives after its key has expired is treated as new.Full article →All terms with the request; the receiver stores the key with the result and returns the stored result on a repeat. Retries without idempotency are not resilience. They are a machine for duplicating side effects.

Running many services: tracing, discovery, gateways and the mesh

With one service, a failed request leaves one stack tracestack tracethe calls in progress at an errorThe list of function calls that were in progress when an error was raised, innermost first, printed with the error. It shows where a failure surfaced, which is not always where it was caused — and a process killed from outside, by the kernel or an orchestrator, leaves none at all.All terms. With many, it leaves fragments in eight sets of logs, and nothing joins them unless you planned for it (row 15). The fix is a trace id created at the edge and passed on every call, so every log line and every timing can be grouped by request. Two standards make this portable. W3C Trace Context, a Recommendation since February 2020, defines the traceparent header that carries the id between services. OpenTelemetryOpenTelemetrya vendor-neutral standard for telemetryAn open-source project that defines how applications produce and send traces, metrics and logs — the libraries, the data format and the protocol — so the tool that stores and shows them can be changed without re-instrumenting the code. It graduated in the Cloud Native Computing Foundation in May 2026. It standardises how telemetry is collected, not what is worth collecting.All terms provides the libraries and the protocol to collect traces, metrics and logs without tying you to one vendor; it graduated in the Cloud Native Computing Foundation on 11 May 2026. Uber described what the absence costs: one investigation meant working through around 50 services across 12 teams to find the root cause.

Service discovery (row 12) is the address book that lets services find each other as instances come and go. On Kubernetes the platform provides it through DNS; elsewhere a registry such as Consul does. Either way, it is now something every call depends on, and it deserves the care of a database. Roblox learned this over 73 hours in October 2021, in one of the five cases below.

A gateway (row 13) gives clients one entry point instead of twenty. Its risk is gravity: authentication, then routing rules, then a little business logic, until the gateway is the monolith again. Lewis and Fowler’s fourth characteristic — smart endpoints, dumb pipes — is the test to hold it to. A useful variant came out of SoundCloud and was written up by Sam Newman in 2015: a backend for each frontend, or BFF. The mobile app and the website each get a thin backend owned by the team that builds that client, so neither client’s needs bend a shared API.

A service mesh (row 14) moves encryption between services, retries, timeouts and metrics out of the code and into a proxy. Classic meshes put the proxy beside each service as a sidecar; Istio’s ambient mode, generally available since version 1.24 in November 2024, runs a shared proxy per node instead. The benefit is uniformity across languages. The cost is a second distributed system to operate and upgrade, and a trap worth naming: if the mesh retries and the application retries too, you have built the multiplying stack from row 2 without writing a line of code. Decide which layer owns retries, and switch them off in the other.

The last row is about change (row 16). A service is independent only if its callers survive its releases. That means changes that stay backward-compatible, a version when they cannot, and a way to know who calls you at all. Consumer-driven contract tests make the last part checkable: each caller publishes the requests it makes and the fields it reads, and the provider’s build fails when a change would break one. The API Design page covers versioning and contracts in depth.

Services and the teams that own them

Microservices are an organisational design as much as a technical one. Melvin Conway observed in 1968 that a system ends up mirroring the communication structure of the organisation that builds it, and the style uses that on purpose: one team, one set of services, one on-call rota. The right size of a service follows from it. It is the size one team can understand, change and run, not a number of lines. And the right number of services is bounded by the number of teams that can own them. The table lists the signals that the two have drifted apart.

Five cases, and what each one was missing

The cases on the Monolith page went back from services to one process. These five stayed with services, and each documents a cost from the table at the top of this page — in the words of the engineers who paid it.

Monzo’s story starts as a security project and ends as a lesson about knowledge. In November 2019 the bank’s security team wanted the service that manages images on savings pots to be unable to talk to a service that moves money. With a handful of services, an allow-list could be kept by hand. Monzo had over 1,500, with more than 9,300 unique connections between them, and the list changed every day. Nobody had it. So the team wrote rpcmap, a tool that reads the Go code of every service and finds the calls it makes to others. It generates one rule file per called service and checks the files on every push, and the owners of critical services can ask to review every new caller. The ledger was locked down first. The lesson reaches beyond security: in a large system of services, the true dependency graph lives only in the code until a tool extracts it, and row 16 cannot be answered without that graph.

Uber is the same problem at a larger size. By July 2020 the company ran around 2,200 critical microservices, and its engineers described the cost plainly. Services had become black boxes whose behaviour could change at any time. Calls went many layers deep, so a latency spike in one dependency cascaded upstream. Finding one root cause had meant working through around 50 services owned by 12 teams. The answer, Domain-Oriented Microservice Architecture, did not merge the services. It grouped them: related services form a domain, domains form layers that limit what may depend on what, and each domain is reached through a single gateway. Uber classified its 2,200 services into 70 domains, and reported that onboarding a new feature onto a platform got 25–50% faster. One more number from the post is worth keeping: Uber measured the half-life of a microservice at 1.5 years. Half of them were replaced every eighteen months, and gateways stopped every replacement from becoming a migration for its callers.

Where it degrades

Microservices fail in public. A monolith rots quietly over a year; a set of services shows its weak spot during the first bad hour. Almost every failure is one of these:

• A shared database, which turns a set of services into a distributed monolith (row 7).
• Services named after technical layers, so every feature touches all of them and nothing releases alone.
• Releases that need several services to go out in a fixed order — the definition of not being independent.
• Long synchronous call chains, where availability multiplies down below what the monolith had.
• Retries at every layer — in the client, the library and the mesh — that multiply into a storm (row 2).
• Retries without idempotency keys, which duplicate side effects under exactly the conditions retries exist for (row 5).
• Save-then-publish code with no outbox, which loses an event every time a process dies at the wrong moment (row 9).
• Sagas with no one asking what a concurrent request sees halfway through (row 8).
• A gateway that has grown business logic, so it is the monolith again — with an extra hop.
• One service per developer, which produces a topology that reflects hiring rather than the domain.
• No distributed tracing, which turns every cross-service incident into archaeology (row 15).

The honest limit is that microservices solve an organisational problem with a technical mechanism. If the constraint is that twelve teams cannot ship without coordinating, the style removes it, and it is worth the bill. If the constraint is that the code is tangled, that deploys are scary or that the database is slow, the style adds a network to the problem and charges for it every month. The network that makes a boundary expensive to cross is also what makes it expensive to move. A wrong boundary in a monolith costs a refactor; between services it costs a migration. And even companies that are good at this end up thinking in bigger units again, as Uber did with its 70 domains.

The practical check takes an afternoon. Pick a recent feature and count the services it touched. Find a release from last month that needed two services deployed together. Check whether every outgoing call has a timeout, and whether retries happen at more than one layer. Search the code for an event published right after a database write without an outbox. Ask who is paged for each service, and whether any service has more owners than one or none. And trace one failed request end to end: if that takes more than ten minutes, row 15 is the first thing to fix.

When to use it

Related pages

• DevelopmentMonolithA monolith is an application that is built, tested and deployed as one unit, whatever its size and however its code is organised inside. Most complaints made about monoliths are really about the missing boundaries inside one — a different problem with a cheaper fix — and a monolith with real module boundaries is the right default for most systems. The page opens with sixteen common complaints and where each one actually lives, then covers the two axes every architecture argument confuses, why one deployable is the default and the best argument against it, drawing and enforcing module boundaries, running a large monolith, where it genuinely runs out, five documented cases, and where it degrades.
• DevelopmentMonolith → MicroservicesMigrating a monolith to microservices means moving an existing system’s capabilities into separately deployed services one at a time, without stopping the system. The migration that works is not a rewrite with a cutover date but a sequence of small extractions behind a facade, each justified by a measured constraint and each with a way back; the hard part is the data. The page opens with one extraction as sixteen steps, then covers what to extract first, six ways to move code without stopping, splitting the data, the model and the teams, running the migration and knowing when to stop, five documented cases, and where it degrades.
• DevelopmentAPI DesignAPI design is the work of deciding what other programs may send to yours, what comes back, and which of those promises you will keep for years. Anything a consumer can observe becomes part of the promise, and you cannot take it back alone. The page opens with sixteen decisions every HTTP API makes, the default to take and what changing each later costs, then covers what the contract includes, methods, status codes and errors, behaviour under retries and load, who may do what, changing a published API, five documented cases, and where it degrades.