The Failure You Can't See
If a server goes down completely, that's almost the easy case. You get a 500, an alert fires, and everyone notices right away.
AI apps fail differently. The server responds just fine. The status code is 200 OK. The screen loads. But what the user actually gets back is an empty string, a raw error message dressed up as an answer, or a response that has nothing to do with what they asked.
Your monitoring dashboard calls this "healthy," because technically, it is. The server responded, and it responded fast. The response just happened to be useless.
Why AI Apps Are Especially Fragile
Most web services draw a fairly clear line between success and failure. When something breaks, it usually shows up as a status code, an exception, or a log entry. AI apps don't work that way. A response can arrive perfectly normally and still need a separate check to see if it's actually valid. Your code can run without a single error while the response it's handling is already broken.
Part of this comes from how AI apps are built. Producing a single answer often means chaining together a vector search, a tool call, and an LLM call, in sequence. If any one link in that chain slips, the whole response comes out wrong in some ambiguous way. In practice, that usually looks like one of a few things:
- The external LLM API returns something strange, just for that one call
- A rate limit error hits, and the app doesn't handle it cleanly — so it shows up as an empty or vague response instead
- A timeout happens, and the app falls back to an empty response or a stale cached value
None of this necessarily shows up in your error logs. As far as the application is concerned, the call succeeded — the external API responded, and a 200 went back to the user. The problem is that what counts as success on your end looks like failure to the person using your app. And these failures don't all look the same. Three patterns show up most often.
3. Three Silent Failures That Normal Monitoring Never Catches
The empty response. A user asks a question, and the answer area shows nothing at all. The server treated an empty string, or a response missing a required field, as a valid result.
The exposed error. An unfiltered error string or stack trace gets rendered straight to the user. The status code still says 200 — but what the user is actually looking at is a debug log.
The response that's lost the thread. Something in the conversation history or prompt gets tangled, and the answer that comes back has nothing to do with the question. Technically, it's a "normal response." To the user, it's just useless.
All three share the same blind spot: none of them are catchable by monitoring that only looks at the HTTP status code. So what should you be looking at instead?
So How Do You Actually Catch This?
The first thing you can do is stop looking only at the status code and start checking the response body itself — whether it's empty, whether it contains a known error string, whether the fields or content you expect are actually there. That covers two of the three failures above: empty responses and exposed errors.
The third — an answer that's technically valid but unrelated to the question — is a harder problem. Catching that usually takes something more like scenario-based testing: sending the same known question on a schedule and checking whether the shape or substance of the answer still holds up.
This general approach is usually called content monitoring or keyword monitoring — checking not just "did the server respond," but "did the user actually get a valid result." Simple to describe. Much less simple to build and run yourself, every time.
We've Been Looking at This Too
We started with the same question when we were designing always-on monitoring: how do you catch a response that's logged as 200 OK, but failed the person on the other end?
What we're building toward is a hidden failure detection feature — checking the response body alongside the status code, aiming to catch the failures that look fine on the surface but aren't.