[PATCH RFC] run-tests: add support for xunit test reports

Augie Fackler raf at durin42.com
Mon Jun 30 11:41:35 CDT 2014


Thanks for the design insights - I'll see about moving things to the custom result class.

On Jun 30, 2014, at 12:37 PM, Gregory Szorc <gregory.szorc at gmail.com> wrote:

> On 6/30/14, 8:13 AM, Augie Fackler wrote:
>> # HG changeset patch
>> # User Augie Fackler <raf at durin42.com>
>> # Date 1403842355 14400
>> #      Fri Jun 27 00:12:35 2014 -0400
>> # Node ID e783e767f540a61eeb877a5b307f1ad76d2d86a6
>> # Parent  becb61de90a1a0384af535a393fb32e7da7a9059
>> run-tests: add support for xunit test reports
>> 
>> The Jenkins CI system understands xunit reports natively, so this will
>> be helpful for anyone that wants to use Jenkins for testing hg.
> 
> I'm overall positive about this patch.
> 
> It's worth noting that Python code for producing xunit results has been written before. Nose - a widely used package that supplements Python's built-in testing mechanisms - has a plugin for this (http://nose.readthedocs.org/en/latest/plugins/xunit.html).

It may interest you to know that I wrote that nose plugin. :)

> My run-tests.py refactoring stopped short of allowing the integration of things like nose into the mix. I'm a bit disappointed you had to write this. But it's not that much code, so meh.

We could (almost certainly) wire up support for .t files as a nose plugin. I'm just lazy, and run-tests is nice because it's in-tree.

> 
>> diff --git a/tests/run-tests.py b/tests/run-tests.py
>> --- a/tests/run-tests.py
>> +++ b/tests/run-tests.py
>> @@ -190,6 +191,8 @@
>>               " (implies --keep-tmpdir)")
>>      parser.add_option("-v", "--verbose", action="store_true",
>>          help="output verbose messages")
>> +    parser.add_option("--xunit", type="string",
>> +                      help="record xunit results at specified path")
>>      parser.add_option("--view", type="string",
>>          help="external diff viewer")
>>      parser.add_option("--with-hg", type="string",
>> @@ -304,6 +307,24 @@
>> 
>>      return log(*msg)
> 
> Both this patch and Anurag's JSON output patch added options for producing extra metadata. I think it would be really nice if machine readable output were enabled by default. We already have enough command arguments as it is.
> 
> That being said, I'm not sure where the logical place for writing this type of output would be.

Yeah, I'm not sure there's any better options.

> 
>> +allsuccess = []
>> +allfailures = {}
>> +def xunitfail(test, lines):
>> +    """Log a failure to the xunit output file if one is in use."""
>> +    allfailures[test.name] = xmlsafe(''.join(lines))
>> +
>> +allerrors = {}
>> +def xuniterr(test, err):
>> +    allerrors[test] = xmlsafe(err)
> 
> I'm not a huge fan of stuffing these in global variables. Can you move these to instance variables of our custom TestResult class?

Sure. That's why I cc'ed you on this - I figured you'd have better design insights than I was getting on my plane home.

> 
>> @@ -1088,8 +1109,14 @@
>> 
>>              self.stream.write('!')
>> 
>> -    def addError(self, *args, **kwargs):
>> -        super(TestResult, self).addError(*args, **kwargs)
>> +    def addSuccess(self, test):
>> +        super(TestResult, self).addSuccess(test)
>> +        allsuccess.append(test)
>> +
>> +    def addError(self, test, err):
>> +        super(TestResult, self).addError(test, err)
>> +        tb = traceback.format_exception(*err)
>> +        xuniterr(test, tb)
> 
> Yes, capturing things on the TestResult instance is the proper approach here.
> 
>>          return accepted
>> @@ -1321,6 +1351,40 @@
>>          for test, msg in result.errors:
>>              self.stream.writeln('Errored %s: %s' % (test.name, msg))
>> 
>> +        if self._runner.options.xunit:
>> +            xuf = open(self._runner.options.xunit, 'wb')
>> +            try:
>> +                timesd = dict(result.times)
>> +                xuf.write(
>> +                    '<?xml version="1.0" encoding="%(encoding)s"?>'
>> +                    '<testsuite name="run-tests" tests="%(total)d" '
>> +                    'errors="%(errors)d" failures="%(failures)d" '
>> +                    'skip="%(skipped)d">\n' % {
>> +                        'total': result.testsRun,
>> +                        'encoding': 'utf-8',
>> +                        'failures': failed,
>> +                        'skipped': skipped + ignored,
>> +                        'errors': 0,
>> +                        })
> 
> Manual XML writing? Can we not use a xml library here?

We could, but last time I wrote this the xml libs turned out to be a pain in the rear. I'll take another look at minidom though.

> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at selenic.com
> http://selenic.com/mailman/listinfo/mercurial-devel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://selenic.com/pipermail/mercurial-devel/attachments/20140630/35aa7dbe/attachment.pgp>


More information about the Mercurial-devel mailing list