/!\ This page is primarily intended for Mercurial's developers.


Plan to use the Hypothesis framework in Mercurial Tests suite.

1. Inline Python test (for .t) file

This made it to core.

2. Doctest testing

Having a way to use fuzzed input in doctest would be interesting. However, Hypothesis does not currently support doctests

3. Side by side comparison

One way Hypothesis could help greatly, is but generating random usage scenario. Being able to ensure that two variants of Mercurial (eg: repo format, extensions) keep the same Behavior would be very handy

For this would need:

3.1. proposal for simple initial step

Aiming at something very simple first would be ideal to have all the hard work in place and let people add complexity as needed.

3.2. More advanced idea

3.3. Current Work In Progres

Current work in progress:

  1. Uses Hypothesis's rule based state machine to generate a sequence of commands against a number of Mercurial repos.
  2. If a command returns a non-zero exit code, the error message is matched against a set of error messages that were expected to be possible there. If it does not match, the test fails.
  3. If the whole sequence of operations successfully completes, they are then rerun against another version of Mercurial for comparison.
  4. Either way, a .t file is generated that captures the behaviour of the executed test (note that this captures the observed behaviour, so the generated .t file is always expected to pass under the current version of Mercurial being tested).

So far the comparison hasn't found anything interesting, but the unexpected error messages test has.

Operations are currently being expressed using @rule() decorators in Python. The currently supported operations are:

CategoryDeveloper CategoryNewFeatures

HypothesisPlan (last edited 2016-02-22 15:05:43 by DavidMacIver)