run-tests.py refactor

Gregory Szorc gregory.szorc at gmail.com
Thu Apr 24 22:03:13 CDT 2014


On 4/24/2014 7:08 PM, Pierre-Yves David wrote:
> 
> 
> On 04/24/2014 02:42 PM, Gregory Szorc wrote:
>> I'm working on a significant refactor of run-tests.py. My work can be
>> found in the indygreg/run-tests-refactor bookmark of
>> https://bitbucket.org/indygreg/hg. I plan to send the patches to this
>> list once the 3.0 window is behind us. But I wanted to give people a
>> heads up in case there are comments before the patch bombing.
> 
> Nice to see people showing interest in the test suite!. Are you aware
> that Anurag Goel will spend his Gsoc working on it? You guys should
> probably talk and synchronize.

I wasn't aware of it until a few days ago, when I was neck deep in my
patch series. I assumed everyone is on this list. Part of the reason I
sent this email today was because I didn't want to jeopardize the GSoC
project and figured getting this patch series resolved and landed sooner
would be best for everyone.

> Cool, where you able to cleanly preserve the parallel run ability?

Of course! I have a custom unittest.TestSuite class whose run() does the
multithreaded foo.

> Did you measured any overhead by using the unittest module (not that I
> expect one, but wondering)?

I didn't notice one. Looking at the unittest implementation, I wouldn't
expect one.

>> The execution time of the Mercurial tests is a common complaint. I did
>> some profiling and determined that the tests were spending an awful lot
>> of time in overhead of invoking the "hg" process. By executing tests in
>> shells, we have to incur the process start-up and repository
>> "re-association" costs for every invocation of "hg." The overhead is
>> significant.
>>
>> I added an experimental "pysh" mode that parses shell commands into
>> Python functions. If a .t file consists of only shell commands that can
>> be inlined to Python, the test executes in pure Python. For "hg" calls,
>> it creates a new mercurial.dispatch.request and calls
>> mercurial.dispatch.dispatch().
> 
> Note that we could also use the command server + chg for that. This
> would have the cool effect to test the commands server too. Have you
> tried in path?

I had considered installing a fake `hg` into the test shell which talked
to a command server. But I haven't implemented it yet.

>>
>> The bad news is that only ~54 of the ~425 existing .t tests can be
>> executed in pure Python. And, even with inlining, total wall time
>> execution for the entire test suite only dropped by 20-30s (i7-2600K - 4
>> + 4 core -j8). The reason we can't inline more tests is because the
>> tests are doing things with the shell that can't yet be parsed into
>> Python functions.
> 
> 20-30 secondes for what total time ?

On my i7-2600K mech HD w/ -l -j8:

Original:

real    6m40.558s
user    31m31.405s
sys     17m48.862s

Mine:

real    6m47.762s
user    31m34.593s
sys     17m56.246s

(within expected deviation)

Mine + --pysh:

real    6m16.644s
user    29m57.186s
sys     16m28.237s

> Could we imagine an hybrid approach? where we stay in python as much as
> possible and fallback to shell when needed?

In theory. But there's all kinds of state you need to account for. You
inevitably have to have something driving the test. You are either
invoking a lot of shells or a lot of python processes. It becomes ugly
fast. You really don't want to be make your test harness a source of
bugs if at all possible. Single mode tests or tests with clear
delimiters for execution domains limit surface area for bugs.



More information about the Mercurial-devel mailing list