New testing framework
Adrian Buehlmann
adrian at cadifra.com
Fri Jun 11 11:07:09 CDT 2010
On 11.06.2010 17:36, Martin Geisler wrote:
> Greg Ward <greg at gerg.ca> writes:
>
>> Hi all --
>>
>> I have been working on a new testing framework in the bfiles extension
>> that is general enough to work for Mercurial itself.
>
> Ah, you too.. :-) I have also made a small framework, and I know Peter
> has as well.
>
> My framework is inspired from doctest: you record a session in your
> terminal and the framework will execute the lines starting with $ and
> compare the output. It looks like this inside reStructuredText:
>
> .. shelltest::
> :name: alice
>
> $ hg version
> Mercurial Distributed SCM (version 1.5)
>
> Copyright (C) 2005-2010 Matt Mackall <mpm at selenic.com> and others
> This is free software; see the source for copying conditions. There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
> (it could also run from a plain file)
>
> The code lives here:
>
> http://bitbucket.org/mg/kick-start/src/tip/shelltest.py
>
> The advantage of this system is that it embeds the output into the
> script. The disadvantage is that it does not make it easier to run the
> test suite on Windows.
It would really love to see a testsuite that's easier to get running on
Windows than
http://mercurial.selenic.com/wiki/WindowsTestingPlan
Quoting: "Few people actually run this procedure and you are likely to
encounter setup issues"
Having to maintain and use a separate patch queue for Windows testing
discourages testing on Windows:
http://hg.intevation.org/mercurial/pmezard/crew-w32.mq/file/tip
>> First, the rationale: i.e. what's wrong with Mercurial's shell
>> script-based testing system:
>> 1) it's not portable -- only works on Unix
>
> Agred, this is a problem.
>
>> 2) the tests are hard to understand, since you have to absorb the
>> shell script and the .out file together
>
> This is also a problem.
>
>> 3) therefore, the tests are hard to modify
>> 4) the tests are slow, because
>> a) every hg command is a separate process (N.B. I do not propose
>> to change this!)
>> b) they're shell scripts, and all that sed'ing and grep'ing means
>> a lot of separate processes
>>
>> Now of course, there are *good* things about the current test system
>> that I don't want to break:
>> 1) it does thorough high-level end-to-end testing
>> 2) run-tests.py does a good job of isolating the test scripts from
>> the surrounding environment
>
> True.
>
>> In the Python version, that becomes:
>>
>> hgt.createstore('test-store')
>> hgt.hg(['clone', '-q', hgt.tjoin('test-repo1.bundle'), 'repo1'],
>> stdout=hgt.ANYTHING)
>> os.chdir('repo1')
>> hgt.hg(['bfupdate'],
>> stdout='5 big files updated, 0 removed\n')
>>
>> And that pretty much sums it up. Highlights:
>>
>> * can't run "tar" portably, so I created a method createstore() that
>> uses the tarfile module
>> * Tester has a dedicated method for running an hg command and
>> verifying its stdout, stderr, and status
>> * passing a list of args to hgt.hg() is annoying, but deeply
>> ingrained in me because I fear and loathe shell quoting rules...
>> however, it might be overkill in this case
>> * test scripts still assume they are being run by run-tests.py, i.e.
>> $TESTDIR, $HGTMP, and $PATH are all set accordingly (but the
>> Tester object generally wraps them so individual scripts don't use
>> the environement vars)
>>
>> The best thing about hgtest is that you don't to diff the output of a
>> script to know that it passed. Just running it with run-tests.py
>> --debug makes failures obvious; e.g. if hg prints "foo" when it should
>> print "bar", hgtest reports this clearly and explicitly.
>
> That sounds like a nice feature -- being able to pin-point the exact
> command that failed per test case.
>
More information about the Mercurial-devel
mailing list