D6122: tests: add a file listing all the flaky tests
lothiraldan (Boris Feld)
phabricator at mercurial-scm.org
Fri Mar 22 05:22:19 EDT 2019
lothiraldan added a subscriber: durin42.
lothiraldan added a comment.
In https://phab.mercurial-scm.org/D6122#89819, @av6 wrote:
> In https://phab.mercurial-scm.org/D6122#89818, @lothiraldan wrote:
> > @av6 is a `--no-flaky-tests` flag what you had in mind?
> Something like that, yes. I suggested that flag seeing how this change introduces a quite specific file with just one line. And I thought "if it's important, let's integrate it deeper into the test runner".
> But a flag for run-tests.py is not the only alternative. Now that I think about `#require`, can test-remotefilelog-bgprefetch.t be marked with `#require not-an-exotic-fs-inside-docker` (or whatever the actual thing that causes flakiness if it's known)? hghave.py already checks various filesystem features, why not make it check for the cause of issue6083?
For this specific test, we are not sure that it's the exact issue, it's our best lead. I'm afraid that identifying the underlying cause will be 90% of the work fixing it.
My goal is to allow people running tests outside our buildbot and gcc build farm to being able to quickly identify which test error are expected or not. And for CI setups, we want the flaky test to not impact the return code as it would makes the build fails in most of CI setups.
Identifying the root cause of the issue is a problem that takes time and expertise, the docker exotic fs lead is actually from @durin42, I would have not guessed it. I would like to find a way for people to report that a test is flaky on their setup without requiring them to dig too deep in the issue.
I would be ok with the `--no-flaky-tests` flag, even if I found the `#require flaky` a bit confusing.
To: lothiraldan, #hg-reviewers
Cc: durin42, av6, mercurial-devel
More information about the Mercurial-devel