New testing framework

[Adrian Buehlmann]

On 12.06.2010 14:22, Peter Arrenbrecht wrote:
> On Fri, Jun 11, 2010 at 9:57 PM, Greg Ward <greg at gerg.ca> wrote:
>> On Fri, Jun 11, 2010 at 12:07 PM, Adrian Buehlmann <adrian at cadifra.com> wrote:
>>> It would really love to see a testsuite that's easier to get running on
>>> Windows than
>>
>> That's my main goal.  (Making it easier to understand and modify the
>> tests is a close second.)  It's working reasonably well; porting the
>> bfiles tests to hgtest actually found a couple of bfiles bugs.
>>
>> One annoyance: I have some test code that is obviously Unix-specific, e.g.
>>
>> if os.name == 'posix':
>>    hgt.announce('bfupdate respects umask')
>>    os.remove('sub/big3')
>>    umask = os.umask(077)
>>    try:
>>        hgt.hg(['bfupdate', 'sub'],
>>               stdout='1 big files updated, 0 removed\n')
>>    finally:
>>        os.umask(umask)
>>    mode = os.stat('sub/big3').st_mode & 0777
>>    hgt.assertequals(0700, mode, 'sub/big3: mode')
> 
> I fully appreciate the need to get the tests running on Windows. But
> the above is simply too hard to read. One of my goals (and I guess
> Martin's) is to have examples in documentation that are actually
> tests. Which would again mean we need readable tests.

I'm not sure how much that above testcase is representative for
readability of the average testcase.

How many of the tests deal with linux specific things like unix file modes?

> So I think that Patrick's Python-based mini-shell coupled with
> something like Matt's or my/Martin's approach is the better way
> forward. My take on this which I use with pbranch is very much like
> Martin's shelltest. But it uses a pipe to a single bash process for
> the entire script (which would be replaced with Patrick's mini-shell).
> This means it could actually keep stdout and stderr apart in the final
> output. (I have not investigated this yet, but my tests run about 3x
> slower on OS X than on Linux. Maybe this is because of the piping).
> With a Python-based mini-shell, we actually would not have to pipe at
> all and could keep hg loaded, so we would probably get speed
> equivalent to Greg's approach.

Windows testing is today probably mostly done by a single person:
Patrick. I've mostly abandoned trying to run the testsuite myself on
Windows. I use an ubuntu in a VM on my Windows 7 if I need to send a
mercurial patch. The rest is manual testing various specific cases when
I'm interested in a specific case. It's simply the line of least resistance.