New testing framework

[Greg Ward]

Hi all --

I have been working on a new testing framework in the bfiles extension
that is general enough to work for Mercurial itself.  I've converted
many of bfiles tests from the familiar shell script style to my new
framework (arrogantly named "hgtest").  I think I've gone far enough
with it that I need some outside feedback: I know what the strengths
and weaknesses of my new framework are, and I'd like your input on how
to make it better.

First, the rationale: i.e. what's wrong with Mercurial's shell
script-based testing system:
  1) it's not portable -- only works on Unix
  2) the tests are hard to understand, since you have to absorb the
shell script and the .out file together
  3) therefore, the tests are hard to modify
  4) the tests are slow, because
    a) every hg command is a separate process (N.B. I do not propose
to change this!)
    b) they're shell scripts, and all that sed'ing and grep'ing means
a lot of separate processes

Now of course, there are *good* things about the current test system
that I don't want to break:
  1) it does thorough high-level end-to-end testing
  2) run-tests.py does a good job of isolating the test scripts from
the surrounding environment

So what's my answer?  Well, obviously, the test scripts are written in
Python.  And it uses a controlling class vaguely inspired by xUnit (it
has methods like asserttrue(), assertequals(), _fail()).  But the
spirit remains much the same: you run a sequence of "hg" commands and
make sure each one did the right thing, ie. wrote what you expected to
stdout and stderr and had the intended side-effects.

Here's an example from bfiles' test suite (test-remove and
test-remove.py).  The old shell script started like this:

  echo "% setup"
  . "$TESTDIR/common"
  createrc

In test-remove.py, that becomes

  import common

  hgt = common.BfilesTester()
  hgt.announce('setup')
  hgt.updaterc()

(common.py contains the bfiles-specific test stuff; the generic
framework is in hgtest.py.  BfilesTester is a bfiles-specific subclass
of hgtest.Tester.)

Back to the shell script, here's something more that occurs in almost
every bfiles test:

  tar -xf $TESTDIR/test-store.tar
  hg clone -q $TESTDIR/test-repo1.bundle repo1
  cd repo1
  hg bfupdate

In the Python version, that becomes:

  hgt.createstore('test-store')
  hgt.hg(['clone', '-q', hgt.tjoin('test-repo1.bundle'), 'repo1'],
          stdout=hgt.ANYTHING)
  os.chdir('repo1')
  hgt.hg(['bfupdate'],
         stdout='5 big files updated, 0 removed\n')

And that pretty much sums it up.  Highlights:

  * can't run "tar" portably, so I created a method createstore() that
uses the tarfile module
  * Tester has a dedicated method for running an hg command and
verifying its stdout, stderr, and status
  * passing a list of args to hgt.hg() is annoying, but deeply
ingrained in me because I fear and loathe shell quoting rules...
however, it might be overkill in this case
  * test scripts still assume they are being run by run-tests.py, i.e.
$TESTDIR, $HGTMP, and $PATH are all set accordingly (but the Tester
object generally wraps them so individual scripts don't use the
environement vars)

The best thing about hgtest is that you don't to diff the output of a
script to know that it passed.  Just running it with run-tests.py
--debug makes failures obvious; e.g. if hg prints "foo" when it should
print "bar", hgtest reports this clearly and explicitly.

The lamest thing is that, because the whole system still relies on
run-tests.py, we end up storing and diffing .out files anyway.
Failures are really ugly and painful in this style; when a test fails,
I just end up running it with run-tests.py --debug, and things are
much clearer.

Anyways: if you're interested, the best thing to do right now is clone
bfiles (http://vc.gerg.ca/hg/hg-bfiles/) and take a good hard look in
the tests/ directory.  I've been deleting the old shell scripts as I
convert them to hgtest, so if you want to do a side-by-side comparison
you'll probably want two working dirs.  Just update to rev
0c979021651a in one to see the shell scripts, then to tip in the
other.

Thanks for any feedback!

Greg