Hypothesis

Plan to use the Hypothesis framework in Mercurial Tests suite.

1. Inline Python test (for .t) file

This made it to core.

2. Doctest testing

Having a way to use fuzzed input in doctest would be interesting. However, Hypothesis does not currently support doctests

3. Side by side comparison

One way Hypothesis could help greatly, is but generating random usage scenario. Being able to ensure that two variants of Mercurial (eg: repo format, extensions) keep the same Behavior would be very handy

For this would need:

• a list of (parameterized) action to be performed randomly.
• a way to check that result is identical
• a way to specify variant

3.1. proposal for simple initial step

Aiming at something very simple first would be ideal to have all the hard work in place and let people add complexity as needed.

• Actions:
  • generate file
  • add/remove/forget/addremove
  • commit
  • pull/push
• Checks:
  • verify (no repo corruption)
  • state of the working copy (hg status)
  • state of the repo (hg log)
• variants:
  • repository format
  • inabling intrusive extension (largefile, watchman, etc

3.2. More advanced idea

• Having more in-depth checks, (eg, compression level),
• Selecting relevant/irrelevant checks for each variants,
• Checking difference in resources consumptions,
• Coverage based example discovery (using cov data to build better scenario)
• Static generation of .t test for high quality scenario,

3.3. Current Work In Progres

Current work in progress:

1. Uses Hypothesis's rule based state machine to generate a sequence of commands against a number of Mercurial repos.
2. If a command returns a non-zero exit code, the error message is matched against a set of error messages that were expected to be possible there. If it does not match, the test fails.
3. If the whole sequence of operations successfully completes, they are then rerun against another version of Mercurial for comparison.
4. Either way, a .t file is generated that captures the behaviour of the executed test (note that this captures the observed behaviour, so the generated .t file is always expected to pass under the current version of Mercurial being tested).

So far the comparison hasn't found anything interesting, but the unexpected error messages test has.

Operations are currently being expressed using @rule() decorators in Python. The currently supported operations are:

• Enable an extension (currently shelve or mq)
• Clone a repo
• Commit
• Pull/Push
• Create/switch branch
• A lot of repo checking options: verify/diff/log/status/etc.
• shelve/unshelve
• addremove
• add/forget/delete

CategoryDeveloper CategoryNewFeatures