buildbot failure in Mercurial on OS X 10.7 hg tests

[Brendan Cully]

On 2012-09-19, at 11:34 PM, Patrick Mézard <patrick at mezard.eu> wrote:

> Le 20/09/12 00:00, Matt Mackall a écrit :
>> On Wed, 2012-09-19 at 08:52 -0700, hgbuildbot at kublai.com wrote:
>>> The Buildbot has detected a new failure on builder OS X 10.7 hg tests while building hg.
>>> Full details are available at:
>>> http://hgbuildbot.kublai.com/builders/OS%20X%2010.7%20hg%20tests/builds/254
>>> 
>>> Buildbot URL: http://hgbuildbot.kublai.com/
>> 
>> I probably stopped looking at these over a month ago because they were
>> too noisy.
>> 
>> In my opinion, the primary benefit of the buildbot (and the test suite
>> in general) is spotting _regressions_. Tests that are unstable (even
>> because of real bugs, like is probably the case here) cause more harm
>> than good here: I'm not going to fix the bug personally and if I have to
>> click through three steps to discover that a particular report is just
>> largefiles flapping in the breeze again, I'm not going to bother and I'm
>> not going to notice real regressions that matter to me. I suspect the
>> same is true for other people around here too.
>> 
>> Similarly, test failures that turn a column red constantly for weeks are
>> also harmful because they'll mask the appearance of new regressions we
>> haven't heard about. The snare has already been sprung, it needs to be
>> reset before it's going to catch any more issues.
>> 
>> Thus, we need the normal, every day state of the buildbot to be "all
>> green" regardless of whether that corresponds with "all tests pass on
>> all systems". The latter isn't all that meaningful anyway, given that we
>> have ~300 open bugs on the BTS to tell us everything is not perfect
>> everywhere anyway.
>> 
>> So what should we do? I think the right thing to do with any flaky or
>> consistently failing buildbot issue is to a) file a bug and b) blacklist
>> the test until some effort has been made to address it so buildbot
>> results can go back to being green.
> 
> +1
> 
> How do we do that? Make a list and give it to Brendan?

I think it's going to work better if we don't have to go through me every time we want to change the list of tests that we expect to pass/fail. Instead, I think we should mark the tests in the repo itself so that buildbot can be configured to ignore them. The blacklist mechanism we already have would probably work fine. It might be nice if we could supply more than one blacklist argument (if we can't already, I haven't checked). That would mean less messing around with tests that already have blacklists.