New testing framework

Adrian Buehlmann adrian at cadifra.com
Fri Jun 11 11:07:09 CDT 2010


On 11.06.2010 17:36, Martin Geisler wrote:
> Greg Ward <greg at gerg.ca> writes:
> 
>> Hi all --
>>
>> I have been working on a new testing framework in the bfiles extension
>> that is general enough to work for Mercurial itself.
> 
> Ah, you too.. :-) I have also made a small framework, and I know Peter
> has as well.
> 
> My framework is inspired from doctest: you record a session in your
> terminal and the framework will execute the lines starting with $ and
> compare the output. It looks like this inside reStructuredText:
> 
> .. shelltest::
>    :name: alice
> 
>    $ hg version
>    Mercurial Distributed SCM (version 1.5)
> 
>    Copyright (C) 2005-2010 Matt Mackall <mpm at selenic.com> and others
>    This is free software; see the source for copying conditions. There is NO
>    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> 
> (it could also run from a plain file)
> 
> The code lives here:
> 
>   http://bitbucket.org/mg/kick-start/src/tip/shelltest.py
> 
> The advantage of this system is that it embeds the output into the
> script. The disadvantage is that it does not make it easier to run the
> test suite on Windows.

It would really love to see a testsuite that's easier to get running on
Windows than

http://mercurial.selenic.com/wiki/WindowsTestingPlan

Quoting: "Few people actually run this procedure and you are likely to
encounter setup issues"

Having to maintain and use a separate patch queue for Windows testing
discourages testing on Windows:

http://hg.intevation.org/mercurial/pmezard/crew-w32.mq/file/tip

>> First, the rationale: i.e. what's wrong with Mercurial's shell
>> script-based testing system:
>>   1) it's not portable -- only works on Unix
> 
> Agred, this is a problem.
> 
>>   2) the tests are hard to understand, since you have to absorb the
>>      shell script and the .out file together
> 
> This is also a problem.
> 
>>   3) therefore, the tests are hard to modify
>>   4) the tests are slow, because
>>     a) every hg command is a separate process (N.B. I do not propose
>>        to change this!)
>>     b) they're shell scripts, and all that sed'ing and grep'ing means
>>        a lot of separate processes
>>
>> Now of course, there are *good* things about the current test system
>> that I don't want to break:
>>   1) it does thorough high-level end-to-end testing
>>   2) run-tests.py does a good job of isolating the test scripts from
>> the surrounding environment
> 
> True.
> 
>> In the Python version, that becomes:
>>
>>   hgt.createstore('test-store')
>>   hgt.hg(['clone', '-q', hgt.tjoin('test-repo1.bundle'), 'repo1'],
>>           stdout=hgt.ANYTHING)
>>   os.chdir('repo1')
>>   hgt.hg(['bfupdate'],
>>          stdout='5 big files updated, 0 removed\n')
>>
>> And that pretty much sums it up.  Highlights:
>>
>>   * can't run "tar" portably, so I created a method createstore() that
>>     uses the tarfile module
>>   * Tester has a dedicated method for running an hg command and
>>     verifying its stdout, stderr, and status
>>   * passing a list of args to hgt.hg() is annoying, but deeply
>>     ingrained in me because I fear and loathe shell quoting rules...
>>     however, it might be overkill in this case
>>   * test scripts still assume they are being run by run-tests.py, i.e.
>>     $TESTDIR, $HGTMP, and $PATH are all set accordingly (but the
>>     Tester object generally wraps them so individual scripts don't use
>>     the environement vars)
>>
>> The best thing about hgtest is that you don't to diff the output of a
>> script to know that it passed.  Just running it with run-tests.py
>> --debug makes failures obvious; e.g. if hg prints "foo" when it should
>> print "bar", hgtest reports this clearly and explicitly.
> 
> That sounds like a nice feature -- being able to pin-point the exact
> command that failed per test case.
> 


More information about the Mercurial-devel mailing list