Merging test scripts and output demo

Greg Ward greg-hg at gerg.ca
Mon Jun 14 07:35:40 CDT 2010


On Sun, Jun 13, 2010 at 2:11 PM, Martin Geisler <mg at lazybytes.net> wrote:
> Though I like the precision of Greg's test framework, I think it's too
> verbose.

Gee, I've been trying to put my finger on what bugs me hgtest as it
stands right now .. is it really as simple as "too verbose"?  Might
be.  Here's what bugs me:

  1) writing "hgt.hg(['clone', '-U', filename])" is a PITA compared to
writing "hg clone -U $filename"
  2) writing
        stdout=('first expected line\n'
                'second expected line\n'
                'third expected line\n')
      is fairly awkward

#1 is easily addressed:

  hgt.hg("clone -U %s" % filename)

I think we can avoid pathological filenames and quoting hell in a
controlled environment like the bfiles or Mercurial test suite.

As for #2, Martin suggests:

> So perhaps we can use Patrick's shell implementation to parse a
> test script into calls to Greg's framework? Something like this:
>
>  % hg pull
>  pulling from ssh://hg@hg.intevation.org/mercurial/crew/
>  searching for changes
>  no changes found
>
> is turned into this call:
>
>  hgt.hg('pull', stdout="""\
>  pulling from ssh://hg@hg.intevation.org/mercurial/crew/
>  searching for changes
>  no changes found""")

OK, that's fine as far as it goes.  And it would probably suffice for,
oh, maybe 40% or 50% of Mercurial's tests.  But as usual, things get
hairy when you want to add some fuzz to the checks or assert that
Mercurial did the right thing rather than merely saying the right
thing:

> As for the portability issues, what if we provide a set of "builtins"
> for the shell-like language we will be executing? So we recognize
> programs like sed, grep, tar, etc. When we write
>
>  % hg pull | sed 's|ssh://.*|<somewhere>|'
>  pulling from <somewhere>
>  searching for changes
>  no changes found
>
> in a test script, it is parsed into code that looks like this:
>
>  p = Pipe()
>  hgt.hg('pull', stdout=p)
>  sed.call('ssh://.*', '<somewhere>', stdin=p, stdout="""\
>  pulling from <somewhere>
>  searching for changes
>  no changes found""")

Gaaaak.  Now you've taken a nice idea and just gone too far.  First,
it's unnecessary, since hgtest is smart enough to do regex matches on
your expected stdout and stderr, e.g. in the bfiles tests I commonly
use

  hgt.hg(['clone', '-U', testbundle, outdir], stdout=hgt.ANYTHING)

where hgt.ANYTHING is defined as

  ANYTHING = re.compile(r'')

Rationale: in bfiles, I don't compare what "hg clone" says.  I'm not
in the business of testing "hg clone", and I assume that it works.
The implementation is fairly obvious: when comparing strings, if the
expected value is a compiled regex, do a regex match rather than a
string comparison.  Work that into your shell-like syntax and the
problem is largely solved.

Another reason why it's not necessary is that hgtest deliberately
mangles the output of hg: specifically, any occurrence of $HGTMP or
$TESTDIR is replaced by the string '$HGTMP' or '$TESTDIR'.  That is,
if hg outputs

   abort: /tmp/hgtest.gn12fs/test-foo/repo1/blah not found

then that will be mangled to

  abort: $HGTMP/test-foo/repo1/blah not found

before it's compared with your expected stderr.  That eliminates
another large reason for sed filtering hackery.

Finally, it's *just* *plain* *wrong*.  Trying to reimplement Bourne
shell by converting a limited subset to Python sounds very painful.
It's approximately as bonkers as, say, compiling Java to JavaScript.
Just because someone has done it doesn't make it a good idea.  ;-)

Now, back to your original idea:

>  % hg pull
>  pulling from ssh://hg@hg.intevation.org/mercurial/crew/
>  searching for changes
>  no changes found

I think the trick is to find a way to add Python code to this.  That
is an excellent shorthand for "run this hg command and assert that its
combined stdout+stderr is X".  I think we can maybe make this an 80%
solution with some way to add Python code around the commands.  Crazy
idea off the top of my head: pre- and post-conditions, e.g.

  % hg pull
  pulling from ssh://hg@hg.intevation.org/mercurial/crew/
  searching for changes
  no changes found
  pre:
    clsize = os.state('.hg/store/00changelog.d').st_size
  post:
    assertequals(clsize, os.state('.hg/store/00changelog.d').st_size)

Hmmm.  This feels much more interesting and creative than the bfiles
tests right now.  Frankly, the hgtest-based bfiles tests are very
plodding, pedestrian, and even clunky.  As you said, "too verbose".

Greg


More information about the Mercurial-devel mailing list