New testing framework

[Matt Mackall]

On Fri, 2010-06-11 at 10:41 -0400, Greg Ward wrote:
> The best thing about hgtest is that you don't to diff the output of a
> script to know that it passed.  Just running it with run-tests.py
> --debug makes failures obvious; e.g. if hg prints "foo" when it should
> print "bar", hgtest reports this clearly and explicitly.

You raise a lot of issues here, I'll try to focus in on just a few.

First, perhaps I'm not getting what's going on here, but I actually
think the diffs we're doing are extremely valuable. We're actually
implicitly and thoroughly testing an awful lot of things that we didn't
set out to test and I'd estimate that the diffs catch something
unrelated to a given test's stated purpose about as often as not.
So I can't say I'm excited about moving away from that. Yes, reading the
diffs can suck sometimes, but I think we can fix that. Also, have you
seen --view?

Second, Python.. Python is a great programming language, but it's a
lousy shell. And that's precisely why it wasn't chosen for the writing
the test suite in the first place (though we've actually supported
running Python scripts from the beginning). Mercurial tests are
currently pretty damn easy to write: many of them are nearly verbatim
from bug reports. I'm not really excited about moving away from that
either, though I see you're giving some thought to making that less
painful.

But I'm also not at all wed to the damned Bourne shell. All I want is
something that looks enough like it so that things that look like
trivial shell scripts (ie 2/3rds of the test suite) continue to work. 
If you can make something that lets us do:

hgt.dumbshell("""
mkdir a
cd a
hg init a
echo a > a
hg ci -Am "this is a test"
hg log
""")

and does the appropriate things (preferably with built-ins) on Windows
and Unix, then you have my attention. 

-- 
Mathematics is the supreme nostalgia of our time.