Testing Mercurial with Hypothesis

[David MacIver]

On 22 February 2016 at 14:24, Pierre-Yves David <
pierre-yves.david at ens-lyon.org> wrote:

>
>>
>> 1. How would you like this integrated, if indeed it is a thing people
>> are interested in seeing in core mercurial at all? Is the prospect of
>> running it as part of run-tests interesting, or would you prefer a
>> different approach? In particular, what should be done with the
>> generated failures?
>>
>
> I think that running them often is valuable.


I agree, but it might be worth thinking about what the ideal workflow of
"running them often" should look like in a Mercurial context.


> We have a "long running" test concept, disabled by default. If we have
> them there and buildbot running them would be nice.
>

One thing I was considering was having them check whether long-running
tests are permitted and adjust their timeout appropriately, as the tests
can be run for a configurable amount of time.


>
> Having some easy way to start 100 exploration in parallel for a couple of
> hours could probably be valuable too.
>

Yeah. At the moment the stateful testing is not *very* well set up for this
usage pattern. In particular it will stop when it finds a single failure.
I'm going to have a think about what the best way to set this up is.


>
> Finally I'm sure we want to make sure any bug it find stay covered. I'm
> not sure if we should write a dedicated manual tests case for each of them
> or rely on hypothesis to save these case and run them during any normal
> run-tests.py.


In general I discourage using the Hypothesis example database for long term
storage. Its intended usage pattern is more that it's a local cache / state
file for development - it ensures that if you find a bug it doesn't go away
until you fix it.

The design of the tests I wrote for Mercurial are that they write a .t file
capturing the behaviour they observed. Turning those into normal Mercurial
tests may be a better way forward.

>
>
> 2. Interesting suggestions for what to test, both specific operations
>> and versions to compare.
>>
>> This is being tracked at
>> https://www.mercurial-scm.org/wiki/HypothesisPlan. I'll try to keep it
>> up to date with any suggestions (it needs a bit of updating already from
>> the current WIP), but feel free to add to it directly.
>>
>
> --
> Pierre-Yves David
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160222/8492c392/attachment.html>