D6122: tests: add a file listing all the flaky tests

lothiraldan (Boris Feld) phabricator at mercurial-scm.org
Fri Mar 22 05:22:19 EDT 2019


lothiraldan added a subscriber: durin42.
lothiraldan added a comment.


  In https://phab.mercurial-scm.org/D6122#89819, @av6 wrote:
  
  > In https://phab.mercurial-scm.org/D6122#89818, @lothiraldan wrote:
  >
  > > @av6 is a `--no-flaky-tests` flag what you had in mind?
  >
  >
  > Something like that, yes. I suggested that flag seeing how this change introduces a quite specific file with just one line. And I thought "if it's important, let's integrate it deeper into the test runner".
  >
  > But a flag for run-tests.py is not the only alternative. Now that I think about `#require`, can ‚Äčtest-remotefilelog-bgprefetch.t be marked with `#require not-an-exotic-fs-inside-docker` (or whatever the actual thing that causes flakiness if it's known)? hghave.py already checks various filesystem features, why not make it check for the cause of issue6083?
  
  
  For this specific test, we are not sure that it's the exact issue, it's our best lead. I'm afraid that identifying the underlying cause will be 90% of the work fixing it.
  
  My goal is to allow people running tests outside our buildbot and gcc build farm to being able to quickly identify which test error are expected or not. And for CI setups, we want the flaky test to not impact the return code as it would makes the build fails in most of CI setups.
  
  Identifying the root cause of the issue is a problem that takes time and expertise, the docker exotic fs lead is actually from @durin42, I would have not guessed it. I would like to find a way for people to report that a test is flaky on their setup without requiring them to dig too deep in the issue.
  
  I would be ok with the `--no-flaky-tests` flag, even if I found the `#require flaky` a bit confusing.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6122

To: lothiraldan, #hg-reviewers
Cc: durin42, av6, mercurial-devel


More information about the Mercurial-devel mailing list