Testing Mercurial with Hypothesis

[Simon Farnsworth]

To give people a sense of what adding a new rule looks like, I've created three commits, which I'll send as replies to this message.

They're published on bitbucket at https://bitbucket.org/simonfar/mercurial-hypothesis/commits/all for reference. I'm running it in David's new "continuous" mode, which runs Hypothesis until a failure occurs – we'll see if it finds the shelve error again.

Simon

From: Mercurial-devel <mercurial-devel-bounces at mercurial-scm.org<mailto:mercurial-devel-bounces at mercurial-scm.org>> on behalf of David MacIver <david at drmaciver.com<mailto:david at drmaciver.com>>
Date: Monday, 22 February 2016 at 14:15
To: "mercurial-devel at mercurial-scm.org<mailto:mercurial-devel at mercurial-scm.org>" <mercurial-devel at mercurial-scm.org<mailto:mercurial-devel at mercurial-scm.org>>
Subject: Testing Mercurial with Hypothesis

Hi there,

I write Hypothesis, a generative testing library for Python. I'm currently being contracted to do some testing of Mercurial using it.

I've currently got a reasonable starting point for this, so this seems like a good time to throw it out into the wild for comments. Do bear in mind that this is very much a work in progress. You can see it at  https://bitbucket.org/david_maciver_/mercurial/src/52e3b0d0fbb4d416b7d7936fd0f88dd61d3a158d/tests/test-verify-repo-operations.py?at=all&fileviewer=file-view-default<https://urldefense.proofpoint.com/v2/url?u=https-3A__bitbucket.org_david-5Fmaciver-5F_mercurial_src_52e3b0d0fbb4d416b7d7936fd0f88dd61d3a158d_tests_test-2Dverify-2Drepo-2Doperations.py-3Fat-3Dall-26fileviewer-3Dfile-2Dview-2Ddefault&d=CwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=mEgSWILcY4c4W3zjApBQLA&m=hAkHxmtWdlVg6k-TxRUBY-tmuEq8INT_NkZMEitOp7I&s=qa-j_rl2MPKQvYEsnEHEvx7w9xgzZWOOrwOduyrBQTQ&e=>

What this does is use Hypothesis's stateful testing (http://hypothesis.readthedocs.org/en/release/stateful.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__hypothesis.readthedocs.org_en_release_stateful.html&d=CwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=mEgSWILcY4c4W3zjApBQLA&m=hAkHxmtWdlVg6k-TxRUBY-tmuEq8INT_NkZMEitOp7I&s=hG7IgXJa5CsUxDeZ3Wy9B-3m4hp6h8Phubdn1YfjMnc&e=>) to run various sequences of mercurial operations against a repo and see if they break. Some "breakages" are acceptable and ignorable, currently identified by what error message Mercurial emits, while others are bugs - either in Mercurial or (more often, for now) in the test.

At the end it generates a file in Mercurial's .t format (using the --interactive mode. I tried writing a generator for this, but getting all the escaping right proved...challenging). If there were no unexpected errors in the run, it then runs that .t test against another version of Mercurial to compare.

The original idea for this testing was to compare different configurations of Mercurial (e.g. pure vs C extensions), though with the design that I've ended up with it also usefully tests a single version of Mercurial as a byproduct.

I'm working with Simon Farnsworth on this, and he ran it over the weekend on his development server and it seems to have produced at least one real bug (attached - Hypothesis normally minimizes better than this, but was running in a suboptimal configuration to do so. We have a much smaller manually minimized example too). I expect as people who have more Mercurial background than I do get involved and find more interesting things for it to test it will become even better at finding them.

I'm open to and interested in comments in general, but I'm particularly interested in:

1. How would you like this integrated, if indeed it is a thing people are interested in seeing in core mercurial at all? Is the prospect of running it as part of run-tests interesting, or would you prefer a different approach? In particular, what should be done with the generated failures?
2. Interesting suggestions for what to test, both specific operations and versions to compare.

This is being tracked at https://www.mercurial-scm.org/wiki/HypothesisPlan<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mercurial-2Dscm.org_wiki_HypothesisPlan&d=CwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=mEgSWILcY4c4W3zjApBQLA&m=hAkHxmtWdlVg6k-TxRUBY-tmuEq8INT_NkZMEitOp7I&s=wwp7xZXERnfcWfRUDw1jc3SIpW9W9ElGHSeTmjoKlC0&e=>. I'll try to keep it up to date with any suggestions (it needs a bit of updating already from the current WIP), but feel free to add to it directly.

Regards,
David

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160222/4c28fa2d/attachment.html>