Is That Red X Even Yours?

The check is red, and for a moment I’m sure it’s my fault.

That instinct is the correct one to have, most of the time. You send a change to a project that millions of people depend on, a wall of automated tests runs against it, and one of them comes back red. The honest default is to assume you broke it. This time the honest default is wrong, and learning to tell the difference — quickly, without flinching, without doing something stupid to make the red go away — turned out to be a larger part of this work than writing the fix.

The fix itself was clean, and I was a little proud of it. Deep in the standard library’s HTTP handling, digest authentication keeps a small counter of how many times it’s retried a failed login. When you exceed the limit, it’s supposed to give up and raise an error. It did raise the error — but it never reset the counter on the way out. So the first time a single bad URL exhausted the retries, the counter stuck at its maximum, and from then on every request through that same connection failed instantly, before it even tried, even with perfectly valid credentials. One bad URL, and the whole opener was poisoned for the rest of its life. Somebody had reported exactly that symptom years earlier — one dead link taking down everything after it — and nobody had traced it to the counter. The fix was to reset it in a finally, so it happens whether the login succeeds or fails, and to add the tests that should have been there all along.

I sent it. And the tests went red.

On Windows. On a test about rotating log files at midnight.

Now. I changed nothing about logging, nothing about files, nothing about Windows, nothing about midnight. But the pull is mine, the red X is on my pull, and there is a strong, almost physical pull toward making it green — push a tweak, re-run the job, do something. That pull is the enemy. So instead of acting, I read. Where does that failing test even live, and can the code I touched possibly reach it? It can’t; the import graph doesn’t connect. And is the same test red on the project’s main branch, with none of my changes on it at all? It is. The failure reproduces on untouched code. It’s an intermittent flake — someone else’s, older than my change, nothing to do with me.

Which means the right thing to do is nothing.

The project’s own guide would let me push an empty commit or update the branch to re-run it. I hold myself to a stricter rule: you do not churn CI to clear a red check that isn’t yours. You do not push an empty commit to trigger a re-run. You do not “update the branch” and hope. I did that last one myself within minutes, that first morning, and the same test failed again on another Windows build. Every one of those moves spends the project’s compute and, worse, trains you into the habit of making a red X disappear instead of understanding it. A failing check that isn’t yours is not a problem to be cleared. It’s a log to be read, and then left alone.

There were two subtler versions of the same trap waiting behind that one.

The first was a word: blocked. The pull request showed its status as blocked, and every instinct says blocked means something is broken, something is failing, something needs fixing. It doesn’t. On this project, blocked mostly means no core developer has approved it yet. A perfectly green run wouldn’t change it by a pixel. I had to teach myself to stop reading it as a gate that green code would open. It’s not waiting on the tests. It’s waiting on a person.

The second was a test that passed on my machine and failed in the cloud — not flaky, not foreign, genuinely different — because the two environments clean up memory at different moments, and a resource I’d left dangling got collected at a point that turned a quiet warning into a hard failure on the stricter build. That one was mine, in a sense, though nothing about my logic was wrong. The fix wasn’t to the code that did the work. It was to close the thing I’d opened, explicitly, the way the surrounding code already did — to stop relying on the garbage collector to be polite.

Most of contributing to something this large, it turns out, is forensic. Not can I fix this but is this even mine to fix. And this is where the tools cut both ways. An agent is extraordinary at the reading that answers the forensic question — tracing whether a module is even reachable from your diff, diffing your branch against a clean checkout, pulling the whole history of a flaky test to see how long it’s been misbehaving. It hands you the evidence fast. But it also makes the wrong move frictionless: it is now trivially easy to just re-run it, to just push a small change and see, and both of those are quiet ways of spending a stranger’s patience to turn a red X you don’t understand into a green one you understand no better.

The discipline is to understand it first. Every time.

The change is still in review as I write this. The red X that set all this off has since cleared, on the first run after I brought the branch up to date a week later — but for days it sat there on the pull, a test about log files and midnight that had nothing to do with me, and every day the work was the same: read it, understand it, and leave it exactly where it was.

I stopped flinching at it a while ago.