New testing framework

[Martin Geisler]

Greg Ward <greg at gerg.ca> writes:

> Hi all --
>
> I have been working on a new testing framework in the bfiles extension
> that is general enough to work for Mercurial itself.

Ah, you too.. :-) I have also made a small framework, and I know Peter
has as well.

My framework is inspired from doctest: you record a session in your
terminal and the framework will execute the lines starting with $ and
compare the output. It looks like this inside reStructuredText:

.. shelltest::
   :name: alice

   $ hg version
   Mercurial Distributed SCM (version 1.5)

   Copyright (C) 2005-2010 Matt Mackall <mpm at selenic.com> and others
   This is free software; see the source for copying conditions. There is NO
   warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(it could also run from a plain file)

The code lives here:

  http://bitbucket.org/mg/kick-start/src/tip/shelltest.py

The advantage of this system is that it embeds the output into the
script. The disadvantage is that it does not make it easier to run the
test suite on Windows.

> First, the rationale: i.e. what's wrong with Mercurial's shell
> script-based testing system:
>   1) it's not portable -- only works on Unix

Agred, this is a problem.

>   2) the tests are hard to understand, since you have to absorb the
>      shell script and the .out file together

This is also a problem.

>   3) therefore, the tests are hard to modify
>   4) the tests are slow, because
>     a) every hg command is a separate process (N.B. I do not propose
>        to change this!)
>     b) they're shell scripts, and all that sed'ing and grep'ing means
>        a lot of separate processes
>
> Now of course, there are *good* things about the current test system
> that I don't want to break:
>   1) it does thorough high-level end-to-end testing
>   2) run-tests.py does a good job of isolating the test scripts from
> the surrounding environment

True.

> In the Python version, that becomes:
>
>   hgt.createstore('test-store')
>   hgt.hg(['clone', '-q', hgt.tjoin('test-repo1.bundle'), 'repo1'],
>           stdout=hgt.ANYTHING)
>   os.chdir('repo1')
>   hgt.hg(['bfupdate'],
>          stdout='5 big files updated, 0 removed\n')
>
> And that pretty much sums it up.  Highlights:
>
>   * can't run "tar" portably, so I created a method createstore() that
>     uses the tarfile module
>   * Tester has a dedicated method for running an hg command and
>     verifying its stdout, stderr, and status
>   * passing a list of args to hgt.hg() is annoying, but deeply
>     ingrained in me because I fear and loathe shell quoting rules...
>     however, it might be overkill in this case
>   * test scripts still assume they are being run by run-tests.py, i.e.
>     $TESTDIR, $HGTMP, and $PATH are all set accordingly (but the
>     Tester object generally wraps them so individual scripts don't use
>     the environement vars)
>
> The best thing about hgtest is that you don't to diff the output of a
> script to know that it passed.  Just running it with run-tests.py
> --debug makes failures obvious; e.g. if hg prints "foo" when it should
> print "bar", hgtest reports this clearly and explicitly.

That sounds like a nice feature -- being able to pin-point the exact
command that failed per test case.

-- 
Martin Geisler

aragost Trifork
Professional Mercurial support
http://aragost.com/mercurial/