The worst bugs return 200
By Panagiotis Athanasakopoulos · Παναγιώτης Αθανασακόπουλος
Engineering14 September 20262 min
A crash is the kind of failure that looks after itself. The page goes blank, an alert fires, somebody notices within minutes and the fix is in by lunchtime. The failures that cost real money are the other kind: the ones that report success. Nothing is red, nothing is logged as an error, and the system goes on doing the wrong thing for as long as nobody looks at the result rather than the status.
They come in a few recognisable shapes. A results page that returns 200 with perfectly valid markup and nothing on it. A nightly job that logs 'completed successfully' every morning while skipping every record that failed validation. An API key that quietly stopped working, and every function that called it treats the missing answer as 'nothing found'. A chatbot apologising that it cannot help is indistinguishable, from the outside, from a chatbot that is right.
Almost all of them are made by code that was trying to be careful. Catch the exception, log it, return an empty list and move on — so that one bad record does not take down the whole page. Each of those decisions is defensible on its own. Together they build a system that converts every failure into a plausible-looking empty answer, and an empty answer is the one thing nobody investigates.
The rule that fixes most of this is to keep 'nothing' and 'could not find out' separate, all the way through. A search that ran and matched no results is an empty list. A search that could not run is an error, and it should travel upwards as one until something that can decide what to do receives it. The two may end up showing the user a similar message; they must never be the same value inside the system.
The second rule is to measure the absence of work. Success is easy to log; it is the missing success that needs counting. How many records did the job touch, against how many it should have? When did this table last receive a row? Which active items were not updated at all in the last run? A job that updates nothing for a week should be as alarming as a job that crashes, and it only will be if someone has written down what normal looks like.
The third is to test the way a person uses the thing, not the way a server reports it. A status code of 200 means the server sent something. It does not mean the page has products on it, that the button in the email leads anywhere useful, or that the form actually saved. The checks worth automating are the ones that follow the link, render the page and count what is there — because that is exactly what the first customer to find the problem will do.
And be careful with caches. A flaky upstream service that occasionally returns nothing, combined with a cache that stores whatever it is given, turns one bad second into hours of confidently empty pages. Never cache an empty result from something you know to be unreliable; let the next request try again.
Loud failures are a gift. The engineering work is to make sure every failure is loud — that nothing in the system is allowed to fail by looking fine.
Common questions
- What is a silent failure in software?
- A failure that does not present as one: the operation reports success or returns an empty but valid response, so no error is raised and no alert fires, while the system is not doing what it should. Silent failures typically persist far longer than crashes because nothing draws attention to them.
- Why is an HTTP 200 response not proof that a page works?
- A 200 status only means the server returned a response. The page can still be empty, show the wrong data or contain broken links. Meaningful checks render the page or follow the link and verify the content a user would actually see.
- How do you monitor a background job that might do nothing?
- Record what each run actually changed and compare it with what it should have changed. Alert when the count of affected records is unexpectedly low or zero, when data has not been updated within its normal interval, or when active items were skipped — not only when the job throws an error.