[PATCH] testing: allow Hypothesis tests to disable extensions

[David MacIver]

Right, what Simon said.

There is something of a question of how useful this is, as it does add a
fair bit of time to the test (your test runner is not fast). I'd probably
advocate for removing the rerunning functionality entirely if after a month
of heavy usage it hasn't found any bugs even though it was sortof the
original point of the exercise.

Having said that, I think the timing issue in shelve was found through
rerunning the final .t test more than once and getting non-deterministic
results, so there's at least some value to it right there.

On 29 February 2016 at 09:00, Simon Farnsworth <simonfar at fb.com> wrote:

> On 29/02/2016, 05:56, "Mercurial-devel on behalf of Martin von Zweigbergk"
> <mercurial-devel-bounces at mercurial-scm.org on behalf of
> martinvonz at google.com> wrote:
>
>
>
> >Pushed to the clowncopter, thanks.
> >
> >On Fri, Feb 26, 2016 at 9:16 AM, David R. MacIver <david at drmaciver.com>
> wrote:
> >> diff -r 18680fed3058 -r 9a28035d39f0
> tests/test-verify-repo-operations.py
> >> --- a/tests/test-verify-repo-operations.py      Wed Feb 24 13:20:06
> 2016 +0000
> >> +++ b/tests/test-verify-repo-operations.py      Fri Feb 26 17:15:49
> 2016 +0000
> >> @@ -190,29 +192,32 @@
> >>          e = None
> >>          if not self.failed:
> >>              try:
> >> -                for ext in (
> >> -                    self.all_extensions - self.non_skippable_extensions
> >> -                ):
> >> -                    try:
> >> -                        os.environ["SKIP_EXTENSION"] = ext
> >> -                        output = subprocess.check_output([
> >> -                            runtests, path, "--local",
> >> -                        ], stderr=subprocess.STDOUT)
> >> -                        assert "Ran 1 test" in output, output
> >> -                    finally:
> >> -                        del os.environ["SKIP_EXTENSION"]
> >>                  output = subprocess.check_output([
> >>                      runtests, path, "--local", "--pure"
> >>                  ], stderr=subprocess.STDOUT)
> >>                  assert "Ran 1 test" in output, output
> >> +                for ext in (
> >> +                    self.all_extensions - self.non_skippable_extensions
> >> +                ):
> >> +                    tf = os.path.join(testtmp,
> "test-generated-no-%s.t" % (
> >> +                        ext,
> >> +                    ))
> >> +                    with open(tf, 'w') as o:
> >> +                        for l in ttest.splitlines():
> >> +                            if l.startswith("  $ hg"):
> >> +                                l = l.replace(
> >> +                                    "--config %s=" % (
> >> +                                        extensionconfigkey(ext),), "")
> >> +                            o.write(l + os.linesep)
> >> +                    with open(tf, 'r') as r:
> >> +                        t = r.read()
> >> +                        assert ext not in t, t
> >> +                    output = subprocess.check_output([
> >> +                        runtests, tf, "--local",
> >> +                    ], stderr=subprocess.STDOUT)
> >> +                    assert "Ran 1 test" in output, output
> >>              except subprocess.CalledProcessError as e:
> >
> >I guess I didn't think of asking on an earlier patch, but why is this
> >rerunning needed? I read something about Hypothesis trying to minimize
> >found examples. I would have thought that that would ideally produce a
> >test case without the call to addextension("unrelatedextension").
>
> The intention here is to check whether an enabled extension changes
> behaviour unexpectedly. For example, we expect that a test case that uses
> watchman to speed up hg status will generate exactly the same results
> (albeit slower) if we run it without watchman enabled. If we see different
> results with watchman enabled, then we've found a bug. This is also why we
> don't disable all extensions; in theory, non_skippable_extensions contains
> the list of extensions that we know are required for this test to run.
>
> Simon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160229/2b38f241/attachment.html>