Design for the failure you had
Resilience engineering usually goes wrong in one of two directions: nothing is built because no failure has happened yet, or a great deal is built against failures nobody has seen and the mechanisms themselves become the largest source of complexity. This estate landed somewhere more disciplined, and it landed there without ever writing the rule down.
Every resilience mechanism in this estate traces to a named incident. That is the doctrine.
Why incident-traced beats threat-modelled, here
The rule is not that imagined failures do not matter. It is a rule about what earns a mechanism, and it produces three properties that speculative resilience work does not:
| Property | Why the incident rule produces it |
|---|---|
| The set stays small | Incidents are rarer than imagined failure modes by orders of magnitude, so the mechanism count stays reviewable. |
| Every mechanism has a story | A new engineer can be told why this exists, with a date. Mechanisms without stories are the ones that get removed by someone tidying up, correctly-looking and wrongly. |
| The tests are real | The failure has occurred, so the reproduction is known. A mechanism against an imagined failure is verified against an imagined reproduction. |
It is the same rule the structural-guard pattern reaches independently — every guard encodes a rule that was violated at least once — which is some evidence that the rule is a real finding rather than a convenient description of what happened to get built.
The patterns
The artefacts themselves — the code, the specifications, the measurements — belong to sg-compute.sgit.ai. This page owns the doctrine, and describes each mechanism only as far as the pattern it demonstrates.
| Mechanism | The pattern it demonstrates |
|---|---|
| The watchdog that exits hard | When the process is deadlocked, an orderly shutdown path is itself deadlocked. The watchdog exits at the lowest level available rather than asking the runtime politely, and the source carries the reasoning about why the polite path cannot work. A recovery mechanism must not depend on the subsystem it is recovering from. |
| The two-phase health poller | Health is not one question. Is it up and is it working have different answers, different latencies, and different correct responses, and collapsing them into one endpoint loses the distinction exactly when it matters. |
| Halt means terminate | A stopped-but-existing resource is a resource that still costs money and can still be reached. The ambiguous middle state is removed rather than managed. |
| Idle reconciliation | Reality drifts from the record whenever anything fails partway. Rather than trying to make every operation atomic, a reconciliation pass runs when nothing else is happening and converges the two. Assume drift; schedule convergence. |
| Build-time guards citing incidents | Each guard names the production incident that motivated it, in the source. That citation is what stops a future cleanup from deleting it. |
Counter-evidence
Three things this doctrine does not do, and one place it failed.
- It cannot cover the failure you have not had yet, and it says so by construction. For a novel or catastrophic first failure, incident-traced resilience is by definition absent. The doctrine is an allocation rule for finite effort, not a safety argument.
- The doctrine was unwritten until this page. Which means it was, until now, a pattern in the artefacts rather than a rule anyone could apply deliberately — and a pattern nobody has stated is one that stops being followed the moment the people who absorbed it move on.
- The mechanisms are not systematically tested against their originating incidents. Each was built in response to a real failure; there is no standing suite that replays those failures to confirm the mechanisms still hold.
- And the honest lapse: a workflow in this estate invokes a binary defined nowhere, sixteen times, and nobody noticed. Resilience machinery does not help with a failure mode nobody is watching for, and the resilient-looking half of a system can coexist with an unread one.