Unified tests and graphlog output

[Adrian Buehlmann]

On 20.09.2010 21:08, Matt Mackall wrote:
> On Mon, 2010-09-20 at 18:48 +0200, Adrian Buehlmann wrote:
>> I found a problem with unified tests containing glog calls (I noticed
>> this while trying to unify tests-rebase*).
>>
>> Assume having the following unified test file (test-test.t):
>>
>>
>>   $ echo "[extensions]" >> $HGRCPATH
>>   $ echo "graphlog=" >> $HGRCPATH
>>   $ hg init
>>   $ echo "foo" > a
>>   $ hg ci -A -m 1
>>   adding a
>>   $ echo "bar" >> a
>>   $ hg ci -m 2
>>   $ hg up 0
>>   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
>>   $ echo "bla" >> a
>>   $ hg ci -m 3
>>   created new head
>>
>>   $ hg glog --template '{rev}:{node|short} "{desc}"\n'
>>   @  2:20c4f79fd7ac "3"
>>   |
>>   | o  1:38f24201dcab "2"
>>   |/
>>   o  0:2a18120dc1c9 "1"
>>
>>
>> This all looks shiny and nice.
>>
>> But...
>>
>> Assume you edit the graphlog output part of that file to:
>>
>>
>>   $ hg glog --template '{rev}:{node|short} "{desc}"\n'
>>   @  2:20c4f79fd7ac "3"
>>   |
>>   | o  1:38f24201dcab "2" EVIL HACKERS WERE HERE
>>   |/
>>   o  0:2a18120dc1c9 "1"
>>
>>
>> Then running this test in a terminal goes like this:
>>
>>
>>   $ python run-tests.py test-test.t -i
>>   .
>>   # Ran 1 tests, 0 skipped, 0 failed.
>>
>>
>> Huh?
>>
>> The test doesn't detect that the command output does not match the
>> expected output?!
>>
>> The symptom is that iff there is a '|' character at the beginning of a
>> line, the rest of the line is irrelevant.
>>
>> Yikes.
>>
>> I haven't looked at the code of the test framework, but my first guess
>> is that we are hit here by the regular expression matching feature of
>> the unified tests.
>>
>> Even more evil: if -- alternatively -- you instead change the graphlog
>> part to:
>>
>>
>>   $ hg glog --template 'HACKED {rev}:{node|short} "{desc}"\n'
>>   @  2:20c4f79fd7ac "3"
>>   |
>>   | o  1:38f24201dcab "2"
>>   |/
>>   o  0:2a18120dc1c9 "1"
>>
>>
>> (note the "HACKED" inserted into the template option)
>>
>> and then accept the changed test file (test-test.t.err) after running
>> that test, you will end having:
>>
>>
>>   $ hg glog --template 'HACKED {rev}:{node|short} "{desc}"\n'
>>   @  HACKED 2:20c4f79fd7ac "3"
>>   |
>>   | o  1:38f24201dcab "2"
>>   |/
>>   o  HACKED 0:2a18120dc1c9 "1"
>>
>>
>> in your "test-test.t" file (note that "HACKED" has -not- been inserted
>> for node 1, because it is behind a '|' !).
>>
>> Can this be improved, so that unified tests are more graphlog-proof?
> 
> Something like this seems to work, I'll probably push it out in a bit.
> 
> diff -r 5f81b1e17787 tests/run-tests.py
> --- a/tests/run-tests.py	Sun Sep 19 13:12:45 2010 -0500
> +++ b/tests/run-tests.py	Mon Sep 20 12:16:11 2010 -0500
> @@ -503,6 +503,9 @@
>  
>      def rematch(el, l):
>          try:
> +            # hack to deal with graphlog
> +            if el.startswith('|'):
> +                el = '\\' + el
>              return re.match(el, l)
>          except re.error:
>              # el is an invalid regex
> 
> 

I see this has been pushed as b016fc1c0862 ("tests: add hack to avoid
problem with graphlog in unified tests").

But b016fc1c0862 only helps for the specific cases reported.

The following (admittedly) contrived hypothetic error in my
'test-test.t' file (from above)


   $ hg glog --template 'HACKED {rev}:{node|short} "{desc}"\n'
   @  HACKED 2:20c4f79fd7ac "3"
   |
   ||o  1:38f24201dcab "2"
   |/
   o  HACKED 0:2a18120dc1c9 "1"


still falsely passes run-tests.py (note the double || in front of
revision 1).

Consider (should fail?)


  $ echo "the | quick | brown | fox | jumps"
  the | quick | BLUE | fox | jumps


which passes run-tests.py (is the fox brown or BLUE?).

And compare with the real life (from test-rebase-cache):


  @  8:c11d5b3e9c00
  |
  o  7:33c9da881988
  |
  | o  6:0e4064ab11a3
  | |
  | o  5:5ac035cb5d8f
  | |
  | | o  4:8e66061486ee
  | | |
  +---o  3:99567862abbe
  | |
  | o  2:65a26a4d12f6
  | |
  | o  1:0f3f3010ee16
  |/
  o  0:1994f17a630e