[PATCH RFC] run-tests: add support for xunit test reports
Augie Fackler
raf at durin42.com
Mon Jun 30 11:41:35 CDT 2014
Thanks for the design insights - I'll see about moving things to the custom result class.
On Jun 30, 2014, at 12:37 PM, Gregory Szorc <gregory.szorc at gmail.com> wrote:
> On 6/30/14, 8:13 AM, Augie Fackler wrote:
>> # HG changeset patch
>> # User Augie Fackler <raf at durin42.com>
>> # Date 1403842355 14400
>> # Fri Jun 27 00:12:35 2014 -0400
>> # Node ID e783e767f540a61eeb877a5b307f1ad76d2d86a6
>> # Parent becb61de90a1a0384af535a393fb32e7da7a9059
>> run-tests: add support for xunit test reports
>>
>> The Jenkins CI system understands xunit reports natively, so this will
>> be helpful for anyone that wants to use Jenkins for testing hg.
>
> I'm overall positive about this patch.
>
> It's worth noting that Python code for producing xunit results has been written before. Nose - a widely used package that supplements Python's built-in testing mechanisms - has a plugin for this (http://nose.readthedocs.org/en/latest/plugins/xunit.html).
It may interest you to know that I wrote that nose plugin. :)
> My run-tests.py refactoring stopped short of allowing the integration of things like nose into the mix. I'm a bit disappointed you had to write this. But it's not that much code, so meh.
We could (almost certainly) wire up support for .t files as a nose plugin. I'm just lazy, and run-tests is nice because it's in-tree.
>
>> diff --git a/tests/run-tests.py b/tests/run-tests.py
>> --- a/tests/run-tests.py
>> +++ b/tests/run-tests.py
>> @@ -190,6 +191,8 @@
>> " (implies --keep-tmpdir)")
>> parser.add_option("-v", "--verbose", action="store_true",
>> help="output verbose messages")
>> + parser.add_option("--xunit", type="string",
>> + help="record xunit results at specified path")
>> parser.add_option("--view", type="string",
>> help="external diff viewer")
>> parser.add_option("--with-hg", type="string",
>> @@ -304,6 +307,24 @@
>>
>> return log(*msg)
>
> Both this patch and Anurag's JSON output patch added options for producing extra metadata. I think it would be really nice if machine readable output were enabled by default. We already have enough command arguments as it is.
>
> That being said, I'm not sure where the logical place for writing this type of output would be.
Yeah, I'm not sure there's any better options.
>
>> +allsuccess = []
>> +allfailures = {}
>> +def xunitfail(test, lines):
>> + """Log a failure to the xunit output file if one is in use."""
>> + allfailures[test.name] = xmlsafe(''.join(lines))
>> +
>> +allerrors = {}
>> +def xuniterr(test, err):
>> + allerrors[test] = xmlsafe(err)
>
> I'm not a huge fan of stuffing these in global variables. Can you move these to instance variables of our custom TestResult class?
Sure. That's why I cc'ed you on this - I figured you'd have better design insights than I was getting on my plane home.
>
>> @@ -1088,8 +1109,14 @@
>>
>> self.stream.write('!')
>>
>> - def addError(self, *args, **kwargs):
>> - super(TestResult, self).addError(*args, **kwargs)
>> + def addSuccess(self, test):
>> + super(TestResult, self).addSuccess(test)
>> + allsuccess.append(test)
>> +
>> + def addError(self, test, err):
>> + super(TestResult, self).addError(test, err)
>> + tb = traceback.format_exception(*err)
>> + xuniterr(test, tb)
>
> Yes, capturing things on the TestResult instance is the proper approach here.
>
>> return accepted
>> @@ -1321,6 +1351,40 @@
>> for test, msg in result.errors:
>> self.stream.writeln('Errored %s: %s' % (test.name, msg))
>>
>> + if self._runner.options.xunit:
>> + xuf = open(self._runner.options.xunit, 'wb')
>> + try:
>> + timesd = dict(result.times)
>> + xuf.write(
>> + '<?xml version="1.0" encoding="%(encoding)s"?>'
>> + '<testsuite name="run-tests" tests="%(total)d" '
>> + 'errors="%(errors)d" failures="%(failures)d" '
>> + 'skip="%(skipped)d">\n' % {
>> + 'total': result.testsRun,
>> + 'encoding': 'utf-8',
>> + 'failures': failed,
>> + 'skipped': skipped + ignored,
>> + 'errors': 0,
>> + })
>
> Manual XML writing? Can we not use a xml library here?
We could, but last time I wrote this the xml libs turned out to be a pain in the rear. I'll take another look at minidom though.
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at selenic.com
> http://selenic.com/mailman/listinfo/mercurial-devel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://selenic.com/pipermail/mercurial-devel/attachments/20140630/35aa7dbe/attachment.pgp>
More information about the Mercurial-devel
mailing list