run-tests.py refactor

[Yuya Nishihara]

On Fri, 25 Apr 2014 14:46:01 -0700, Gregory Szorc wrote:
> On 4/25/2014 6:10 AM, Martin Geisler wrote:
> > Gregory Szorc <gregory.szorc at gmail.com> writes:
> > 
> >> On my i7-2600K mech HD w/ -l -j8:
> >> 
> >> Original:
> >> 
> >> real    6m40.558s user    31m31.405s sys     17m48.862s
> > 
> > I normally run with -j 20 and get a real time of 4m24s. I find that
> > there's so much IO going on that I can start that many processes in
> > parallel to really saturate the CPU (maybe -j 15 would be the
> > same).
> 
> I created a tmpfs to isolate the impact of I/O from my mechanical HD.
> My 4+4 cores were largely saturated during -j8. There were only brief
> periods (maybe ~20s total) where idle CPU was below 5%. Raising it to
> -j10 made the lulls go away.
> 
> Total execution time on the tmpfs was about the same as the mechanical
> HD. I can only conclude that the tests are CPU bottlenecked. My
> previous measurements indicate the bulk of that is inside `hg`.
> 
> I may very well write a feature that invokes the Python profiler on
> all the `hg` invocations and sums the results. Could be interesting.

IIRC, slow module lookup of Python contributes to the startup time of `hg`
process.  It might be partly an I/O bottleneck.

Regards,