buildbot failure in Mercurial on OS X 10.7 hg tests

[Patrick Mézard]

Le 20/09/12 00:00, Matt Mackall a écrit :
> On Wed, 2012-09-19 at 08:52 -0700, hgbuildbot at kublai.com wrote:
>> The Buildbot has detected a new failure on builder OS X 10.7 hg tests while building hg.
>> Full details are available at:
>>  http://hgbuildbot.kublai.com/builders/OS%20X%2010.7%20hg%20tests/builds/254
>>
>> Buildbot URL: http://hgbuildbot.kublai.com/
> 
> I probably stopped looking at these over a month ago because they were
> too noisy.
> 
> In my opinion, the primary benefit of the buildbot (and the test suite
> in general) is spotting _regressions_. Tests that are unstable (even
> because of real bugs, like is probably the case here) cause more harm
> than good here: I'm not going to fix the bug personally and if I have to
> click through three steps to discover that a particular report is just
> largefiles flapping in the breeze again, I'm not going to bother and I'm
> not going to notice real regressions that matter to me. I suspect the
> same is true for other people around here too.
> 
> Similarly, test failures that turn a column red constantly for weeks are
> also harmful because they'll mask the appearance of new regressions we
> haven't heard about. The snare has already been sprung, it needs to be
> reset before it's going to catch any more issues.
> 
> Thus, we need the normal, every day state of the buildbot to be "all
> green" regardless of whether that corresponds with "all tests pass on
> all systems". The latter isn't all that meaningful anyway, given that we
> have ~300 open bugs on the BTS to tell us everything is not perfect
> everywhere anyway.
> 
> So what should we do? I think the right thing to do with any flaky or
> consistently failing buildbot issue is to a) file a bug and b) blacklist
> the test until some effort has been made to address it so buildbot
> results can go back to being green.

+1

How do we do that? Make a list and give it to Brendan?

--
Patrick Mézard