Most software never has to answer this question. A photo app that crashes is a photo app that crashed; nothing else happens. A security product is different, because it sits between the customer and the thing they are trying to do. When it breaks, one of two things must follow: either the customer keeps going without the protection, or the customer stops. There is no third option, and the one you did not choose is the one you will meet by accident, sooner or later.
The product from post 4 — phishing protection for subscribers of a European mobile operator — is a DNS filter that lives inside a VPN tunnel on the phone. Every name the phone looks up passes through it. Which means that if the filter has a bad day, the phone's internet could have one too — unless we decide otherwise, in advance, and build the decision in.
We decided in April. The library changelog has a section I am fond of, titled Deliberate non-choices, and one entry in it reads — product name aside — “No includeAllNetworks on On-Demand — would block all internet when tunnel down (the app is fail-open).” Four words in parentheses carry the whole product philosophy. A filter that took a subscriber's phone offline every time the tunnel hiccuped would not stay installed for long, and a filter that is not installed protects nobody. So: when we break, the customer keeps going. Fail open.
Fail-open is not one setting; it is a chain, and every link in it was written as an explicit rule. Empty block lists allow everything — never block everything, says the matcher's documentation. If the dedicated encrypted resolver cannot be reached, the phone falls through to the system resolver, and from there to public anycast resolvers as the outermost shell. And the part a bug taught us: answers from those fallback tiers are never cached. The fallback resolver is not the authoritative threat-intelligence source; it is the thing you talk to because the real source is down. Cache its “allow” for a blocked domain and you would quietly weaken your own filter for the length of a TTL, even after the real resolver is back. The test that pins this down says so in its own comment: the outermost fail-open shell must not write to the cache.
So by June the direction was chosen, written, tested. Then a phone did something we had not modelled — as phones do.
One subscriber's phone, one evening, changing networks between Wi-Fi and cellular twenty-six times in four minutes. Each handover pulled the underlying network out from under the encrypted DNS transport. The telemetry that reached us the next morning: 1,191 in-tunnel resolver failures in two minutes — 20:40 to 20:41 UTC — and 1,303 within the hour. The existing remedy, which evicted the HTTP connection pool after three consecutive failures, fired hundreds of times without helping: a fresh connection to a network interface that no longer exists is still a connection to nowhere.
And here is the point of the post. The design said fail-open. The subscriber saw DNS_PROBE_FINISHED_BAD_CONFIG on github.com and sorted it out by toggling Wi-Fi. For that person, for those minutes, a fail-open product behaved as if it were fail-closed. Not because anyone had changed the decision — because the failure arrived through a door the decision had not covered. The fallback chain assumed the transport was up but unhappy. It was not up. It was bound to a network that had already gone.
The fleet's first response landed the next day at 12:12, commit c799213. Three of its four parts have stayed with us. A sliding-window storm detector — thirty-second window, ten failures, sixty-second cool-down, all three numbers calibrated on that phone's telemetry rather than guessed. The failure event gained the actual error string, bounded to 200 characters so that a sustained storm could not bloat the log it was supposed to explain. A new tunnel.doh_storm event, emitted once per crossing, so a storm would be one line in the record and not a thousand.
The fourth part wired the detector to a full tunnel restart. Detect the storm, tear the tunnel down, bring it back up. It is the intuitive move — when something is stuck, restart it — and it is the kind of move that looks right until you measure it. By that evening the same phone's telemetry had: twenty restarts initiated, twenty started_no_confirm, zero tunnel.up. Stop-then-start on Android reused the live service instance; the start path saw a service already running and returned early; no new tunnel was ever established. And twenty rapid stop/start cycles on a process already under network stress were a plausible contributor to a second set of failures seen in the same window. The remedy was not helping, and may have been adding to the load.
The morning brought two changes, and the order matters. At 07:32, 3eaf25c fixed the actual cause. The socket had been protected — exempted from the VPN's own routing so the encrypted DNS would not loop back into the tunnel — but protection is not pinning. When Android swapped its default network to a freshly attached interface that had not yet been validated, the protected socket followed the default and had no route: ENETUNREACH, 960 times in one session. The fix tracks the underlying network only once the system marks it validated, binds the DNS socket to that network explicitly, and parks lookups during a handover instead of firing them into the gap.
At 10:01, 438d33e7 quietly took the restart out of the loop. Not the detector — the detector stays, watching, still writing its one line per storm. Only the wire from “storm detected” to “restart the tunnel” was cut. The restart function itself was left in the codebase, on the reasoning that other callers might one day want a corrected version and that deleting it was premature. And the tests were not deleted either. The two that had asserted a restart happened were flipped to assert that on the storm path no restart happens; three unit tests for the restart function were marked ignored with a pointer back to the rollback rationale. Anyone who opens that file next spring will find not an absence but an explanation.
The lesson is not “restarts are bad.” It is that a direction of failure is a promise, and promises are tested in the places you did not think to look. We had written fail-open into the on-demand rules, the matcher, the resolver tiers, the cache. We had not written it into the socket layer, because nobody had asked the socket layer which network it was standing on. The storm simply found the one place the sentence had not reached yet.
Three things held, and they are the same three that held in every incident in this series. The decision was written, so the subscriber's evening was a bug against a stated intent and not a debate about what the product should do. The telemetry was rich enough to grade our own fix within hours — including the part that did not help, which is the part you most need to see. And the retreat was itself recorded: a rollback with a reason, tests inverted rather than removed, a dead function left standing with a note on it. A change that cannot be traced back to a decision is not really a fix yet; it is a hope that happens to compile.
The rule this incident taught me: choose which way you fall before you fall, write it where the next session will read it — and then go looking for the layer that never heard the decision. The fleet that ships this product makes the opposite choice for itself, and for the same reason: an agent's reach is denied by default, and every refusal is written down. Fail-open for the customer, fail-closed for the machine that serves them. Both are only worth anything if you can show, afterwards, that the direction was chosen and not stumbled into. The protocol makes the choosing a written step.
Next post: the closing card of the series — five posts, one lesson each, and what the notes become once the fleet moves on to building Saphan Studio.