buildbot failure in Mercurial on OS X 10.7 hg tests

Matt Mackall mpm at selenic.com
Thu Sep 20 11:26:52 CDT 2012


On Wed, 2012-09-19 at 23:41 -0700, Brendan Cully wrote:
> On 2012-09-19, at 11:34 PM, Patrick Mézard <patrick at mezard.eu> wrote:
> 
> > Le 20/09/12 00:00, Matt Mackall a écrit :
> >> On Wed, 2012-09-19 at 08:52 -0700, hgbuildbot at kublai.com wrote:
> >>> The Buildbot has detected a new failure on builder OS X 10.7 hg tests while building hg.
> >>> Full details are available at:
> >>> http://hgbuildbot.kublai.com/builders/OS%20X%2010.7%20hg%20tests/builds/254
> >>> 
> >>> Buildbot URL: http://hgbuildbot.kublai.com/
> >> 
> >> I probably stopped looking at these over a month ago because they were
> >> too noisy.
> >> 
> >> In my opinion, the primary benefit of the buildbot (and the test suite
> >> in general) is spotting _regressions_. Tests that are unstable (even
> >> because of real bugs, like is probably the case here) cause more harm
> >> than good here: I'm not going to fix the bug personally and if I have to
> >> click through three steps to discover that a particular report is just
> >> largefiles flapping in the breeze again, I'm not going to bother and I'm
> >> not going to notice real regressions that matter to me. I suspect the
> >> same is true for other people around here too.
> >> 
> >> Similarly, test failures that turn a column red constantly for weeks are
> >> also harmful because they'll mask the appearance of new regressions we
> >> haven't heard about. The snare has already been sprung, it needs to be
> >> reset before it's going to catch any more issues.
> >> 
> >> Thus, we need the normal, every day state of the buildbot to be "all
> >> green" regardless of whether that corresponds with "all tests pass on
> >> all systems". The latter isn't all that meaningful anyway, given that we
> >> have ~300 open bugs on the BTS to tell us everything is not perfect
> >> everywhere anyway.
> >> 
> >> So what should we do? I think the right thing to do with any flaky or
> >> consistently failing buildbot issue is to a) file a bug and b) blacklist
> >> the test until some effort has been made to address it so buildbot
> >> results can go back to being green.
> > 
> > +1
> > 
> > How do we do that? Make a list and give it to Brendan?
> 
> I think it's going to work better if we don't have to go through me
> every time we want to change the list of tests that we expect to
> pass/fail.

Indeed.

>  Instead, I think we should mark the tests in the repo itself so that
> buildbot can be configured to ignore them. The blacklist mechanism we
> already have would probably work fine. It might be nice if we could
> supply more than one blacklist argument (if we can't already, I
> haven't checked). That would mean less messing around with tests that
> already have blacklists.

We should probably take advantage of the fact that a bunch of folks have
login access on mercurial.selenic.com and put the control files in a
shared directory there so that the buildbot can wget them. We can
perhaps have a format like:

blacklist/hg-stable:
test-color.t # consistently fails on py2.4 3627
test-largefiles.t # unstable output 3628

..and possibly automatically remove/comment the entries when bugs get
fixed.



-- 
Mathematics is the supreme nostalgia of our time.




More information about the Mercurial-devel mailing list