On this page
During a production incident on an AI-agent platform I work on as fractional CTO — the Worker was being killed for exceeding its memory limit, thousands of times — I produced three root-cause hypotheses in a row. Each one was plausible. Each one had supporting log lines. Each one was wrong.
The bug wasn’t in my reasoning. It was in my dataset. All three conclusions came from the observability dashboard’s log export, which silently caps at 2,000 rows. This post is about that class of mistake, because I don’t think I’m the only one making it.
Three confident, wrong conclusions
Hypothesis one: large payloads. The export showed memory kills clustered around endpoints returning big responses. Obvious story: big response, big buffer, dead isolate. We spent time on payload trimming that changed nothing.
Hypothesis two: the image-serving surface. The sample contained kills on image content routes. Also a clean story — images are big, images kill workers. When the full census finally arrived, that surface accounted for 4 kills out of 11,058. It streams from object storage and never buffers; it was never the problem. We had ranked it near the top.
Hypothesis three: one heavy user. The sample showed one account appearing over and over around the crashes. The story wrote itself: someone is hammering us. In reality that user was simply present in the biased window we happened to export — the actual driver was a frontend render loop that any user of one particular page would trigger.
Three hypotheses, three plausible narratives, zero correct. And note what they had in common: each one suggested a fix that would have consumed days and moved nothing.
What the full window showed
The fix for the dataset was to stop using the dashboard export and pull the complete history through the observability API — every event, the whole retention window, scripted.
The full census: 11,058 kills over 7 days, spread across 15 client locations and 12 projects. The ranking it produced was nearly the inverse of the sampled one. Nearly half of all kills came from one realtime-authorization surface that had barely registered in the export. The “obvious” image surface was statistical noise. The fixes that followed from the census produced a 99.5% reduction within a day.
Two of our early conclusions came from a truncated export and were wrong. I now treat that sentence as the incident’s real lesson — more than anything about memory.
Why truncated data is worse than no data
No data forces you to go get data. Truncated data hands you a dataset that looks complete, engages all your pattern-matching, and rewards you with a coherent story. The 2,000-row cap didn’t announce itself; the export just… ended. Nothing in the file said “this is 2% of the events.”
Sampling isn’t inherently the problem — statisticians sample all the time. The problem is unintentional, biased sampling: an export capped by recency or by whatever ordering the dashboard uses gives you a window correlated with time of day, deploy timing, or one user’s session. Every correlation in that window is a candidate false lead.
The second trap: filtering by severity
There was a subtler version of the same mistake waiting behind the first one. The runtime’s isolate memory gauges were logged at info level. An errors-only filter — the natural thing to reach for during an incident — could not validate or refute a memory claim at all, because the evidence wasn’t classified as an error.
Severity filters encode someone else’s opinion about what matters. During an incident, that opinion is frequently wrong.
The rules I took away
- Before optimizing anything, confirm you have the whole window. Ask explicitly: what is the row cap, the retention limit, the sampling rate of this view? If you can’t answer, you don’t have a census, you have an anecdote.
- Script the pull. Dashboards are for browsing; APIs are for measuring. We ended up with a small script that pulls the full observability window and it has paid for itself several times over.
- Rank by counted totals, not by what’s visible. A priority order built from a sample inverted our real priorities. The count is cheap once you have the full data — get it before assigning work.
- Don’t filter by severity when hunting a mechanism. The signal you need may be logged as routine telemetry.
- Treat a too-clean narrative as a smell. All three wrong hypotheses were better stories than the truth. The truth — a dependency array in a React component — would never have made it through narrative selection.
None of this is specific to Cloudflare or to memory. Anywhere a console exports “the logs” — an APM tool, a hosting dashboard, a database slow-query view — there is a cap, and the cap is rarely written where you’ll see it. The first debugging question isn’t “what does the data say?” It’s “is this all of the data?”