Development of a performance tracking tool for Mercurial

David Douard david.douard at logilab.fr
Wed Mar 30 04:21:16 EDT 2016


On 03/30/2016 09:52 AM, David Douard wrote:
> On 03/30/2016 12:55 AM, Martin von Zweigbergk wrote:
>>
>>
>> On Tue, Mar 29, 2016 at 3:21 PM Pierre-Yves David <pierre-yves.david at ens-lyon.org <mailto:pierre-yves.david at ens-lyon.org>> wrote:
>>
>>
>>
>>     On 03/22/2016 01:54 AM, David Douard wrote:
>>     > Hi everyone,
>>     >
>>     > we (Philippe and a bit of myself, at Logilab) are beginning to work on a
>>     > performance tracking system for Mercurial.
>>     >
>>     > The need for such a tool has been expressed by Pierre-Yves who managed to
>>     > get the project financed by fb.
>>     >
>>     > We've started (mostly Philippe did) by studying several solutions
>>     > to start from.
>>     >
>>     > Below is a "quick" report on what we did for now.
>>     >
>>     > There is an html version of this document on
>>     >
>>     >    https://hg.logilab.org/review/hgperf/raw-file/tip/docs/tools.html
>>
>>     Thanks for posting this, I've inlined the document in my reply for
>>     easier comment.
>>
>>     > #####################################################
>>     > Mercurial performance regression detection and report
>>     > #####################################################
>>     >
>>     > Objectives
>>     > ==========
>>     >
>>     > Mercurial code change fast and we must detect and prevent performances
>>     > regressions as soon as possible.
>>     >
>>     > * Automatic execution of performance tests on a given Mercurial revision
>>     > * Store the performance results in a database
>>     > * Expose the performance results in a web application (with graphs, reports, dashboards etc.)
>>     > * Provide some regression detection alarms with email notifications
>>     >
>>     > Metrics
>>     > ~~~~~~~
>>     >
>>     > We already have code that produce performance metrics:
>>     >
>>     > * Commands from the perf extension in contrib/perf.py
>>     > * Revset performance tests contrib/revsetbenchmarks.py
>>     > * Unit test execution time
>>     > * Annotated portions of unit test execution time
>>
>>
>> I don't think we have many unit tests, and I don't think it's worth timing them anyway. The numbers would change too easily due to simple refactorings (e.g. moving code into or out of the tested function). Perhaps you mean the test run by run-tests.py. They're more relevant but would of course change whenever we add or remove tests to them (that may not be so bad).
>>  
>>
> 
> We are thinking of tracking execution time for each unit test (means, each test run via 
> run-tests.py, yes, but individually, not as a whole).
> But we also plan to detect modifications of the test itself to prevent from
> sending regression notifications in the case where the failing test test itself 
> (in terms of execution time) is modified in the offending changeset. 

Just to make things clear, when I say "unit test", one should read "automatic test" (I don't care a lot about 
what "unit test" means in general, since I believe there is no such thing as unit test really).

Sorry for the confusion, if any.


David


-- 

David DOUARD		 LOGILAB
Directeur du département Outils & Systèmes

+33 1 45 32 03 12	 david.douard at logilab.fr
       	                 http://www.logilab.fr/id/david.douard

Formations - http://www.logilab.fr/formations
Développements - http://www.logilab.fr/services
Gestion de connaissances - http://www.cubicweb.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: david_douard.vcf
Type: text/x-vcard
Size: 302 bytes
Desc: not available
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160330/7990c51a/attachment-0001.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160330/7990c51a/attachment-0001.sig>


More information about the Mercurial-devel mailing list