[PATCH 0 of 5] Add 'unified' tests to the test framework

[Nicolas Dumazet]

On Tue, 03 Aug 2010 11:13:17 -0500
Matt Mackall <mpm at selenic.com> wrote:

> On Tue, 2010-08-03 at 08:56 +0200, Dirkjan Ochtman wrote:
> > On Tue, Aug 3, 2010 at 07:15, Matt Mackall <mpm at selenic.com> wrote:
> > > This is an experiment in unifying test commands and results as
> > > discussed at the 1.5 sprint. See the last three patches for what tests
> > > look like in this model. It's quite an improvement. And when something
> > > breaks, you get to see the changed output in the context of the
> > > commands that caused it. Also, there's some basic support for
> > > filtering of output with regexes, like this:
> > >
> > > Strip 1: expose an old head:
> > >
> > >  $ hg --config extensions.mq= strip 5
> > >  saved backup bundle to .*
> > >
> > > ..which means a lot of the hoops tests jump through right now with sed
> > > and grep are no longer needed.
> > 
> > It looks very nice! Definitely something I'd use in other projects as well.
> > 
> > One thing that I think could be better is the need for indentation
> > before $ hg command. I understand that we need something to
> > disambiguate comments from command output, but I feel that doing it
> > the other way around would be better (prefix the comments, not the
> > output). Primarily because I think it's easier to cut-and-paste
> > command output that way, and we probably have more command output than
> > comments anyway.
> 
> There's not much need to cut and paste command output though. You simply
> write the test without output, then run it, and you end up with the
> unified result in .err. And you either copy that over, or use -i to let
> run-tests do it for you. In other words, just like you'd write a normal
> test.

I just pushed a few converted tests. The new format is definitely an
improvement. I had to debug a few mistakes of mine, and working with
tests just got much more efficient... Really, just give it a try, folks ;)

> 
> >  We could prefix comments with something like # or %.
> > I guess that introduces a chance of ambiguity for output starting with
> > # or %, but we could mitigate that by having a longer prefix, perhaps?
> > (e.g. %%, although I think % is sufficiently rare in our test output
> > and/or should be easy to prevent in most cases).

I'm happy with the current format. Your editor should not have much troubles
catching up with identation after the first "  $ " line. And it feels
really natural, as it's close to the syntax you usually use to describe
shellscript and output in bugtracker/pastebin/...


The only issue that I found so far was with commands that do not include an
EOL. echo -n, and cat of files without EOL is broken, because the output will
mix the echo'ed content and the following SALT:

-  $ echo -n "foobar"
+  foobarSALTxxxxxxxxx.xx YYY

But it should not be too hard to fix in tests. I used "cat filewithouteol; echo"
instead of "cat filewithouteol" in test-tag, for example.
I dont think that we have clean usecases for echo -n anyway, unless it's something
like:

  $ foo() {
  >   echo -n "doing blablabla ..."
  >   blabla && echo "ok" || echo "failed"
  > }

and it will work with current implementation/forces us to use a small subfunction, which
looks cleaner to me.

Regards,
-- 
Nicolas Dumazet — NicDumZ