buildbot failure in Mercurial on OS X 10.7 hg tests

[Matt Mackall]

On Wed, 2012-09-19 at 08:52 -0700, hgbuildbot at kublai.com wrote:
> The Buildbot has detected a new failure on builder OS X 10.7 hg tests while building hg.
> Full details are available at:
>  http://hgbuildbot.kublai.com/builders/OS%20X%2010.7%20hg%20tests/builds/254
> 
> Buildbot URL: http://hgbuildbot.kublai.com/

I probably stopped looking at these over a month ago because they were
too noisy.

In my opinion, the primary benefit of the buildbot (and the test suite
in general) is spotting _regressions_. Tests that are unstable (even
because of real bugs, like is probably the case here) cause more harm
than good here: I'm not going to fix the bug personally and if I have to
click through three steps to discover that a particular report is just
largefiles flapping in the breeze again, I'm not going to bother and I'm
not going to notice real regressions that matter to me. I suspect the
same is true for other people around here too.

Similarly, test failures that turn a column red constantly for weeks are
also harmful because they'll mask the appearance of new regressions we
haven't heard about. The snare has already been sprung, it needs to be
reset before it's going to catch any more issues.

Thus, we need the normal, every day state of the buildbot to be "all
green" regardless of whether that corresponds with "all tests pass on
all systems". The latter isn't all that meaningful anyway, given that we
have ~300 open bugs on the BTS to tell us everything is not perfect
everywhere anyway.

So what should we do? I think the right thing to do with any flaky or
consistently failing buildbot issue is to a) file a bug and b) blacklist
the test until some effort has been made to address it so buildbot
results can go back to being green.

-- 
Mathematics is the supreme nostalgia of our time.