run-tests.py refactor

[Pierre-Yves David]

On 04/24/2014 08:03 PM, Gregory Szorc wrote:
> On 4/24/2014 7:08 PM, Pierre-Yves David wrote:
>>
>>
>> On 04/24/2014 02:42 PM, Gregory Szorc wrote:
>>> I'm working on a significant refactor of run-tests.py. My work can be
>>> found in the indygreg/run-tests-refactor bookmark of
>>> https://bitbucket.org/indygreg/hg. I plan to send the patches to this
>>> list once the 3.0 window is behind us. But I wanted to give people a
>>> heads up in case there are comments before the patch bombing.
>>
>> Nice to see people showing interest in the test suite!. Are you aware
>> that Anurag Goel will spend his Gsoc working on it? You guys should
>> probably talk and synchronize.
>
> I wasn't aware of it until a few days ago, when I was neck deep in my
> patch series. I assumed everyone is on this list. Part of the reason I
> sent this email today was because I didn't want to jeopardize the GSoC
> project and figured getting this patch series resolved and landed sooner
> would be best for everyone.

Good move, having you working on the same area than a Gsoc student seem 
a win as you can probably collaborate on it.

>
>> Cool, where you able to cleanly preserve the parallel run ability?
>
> Of course! I have a custom unittest.TestSuite class whose run() does the
> multithreaded foo.

Multithreading? So each TestCase is pawning a new process running the 
actual test on the *.t file and and wait for it to return?

>>> The execution time of the Mercurial tests is a common complaint. I did
>>> some profiling and determined that the tests were spending an awful lot
>>> of time in overhead of invoking the "hg" process. By executing tests in
>>> shells, we have to incur the process start-up and repository
>>> "re-association" costs for every invocation of "hg." The overhead is
>>> significant.
>>>
>>> I added an experimental "pysh" mode that parses shell commands into
>>> Python functions. If a .t file consists of only shell commands that can
>>> be inlined to Python, the test executes in pure Python. For "hg" calls,
>>> it creates a new mercurial.dispatch.request and calls
>>> mercurial.dispatch.dispatch().
>>
>> Note that we could also use the command server + chg for that. This
>> would have the cool effect to test the commands server too. Have you
>> tried in path?
>
> I had considered installing a fake `hg` into the test shell which talked
> to a command server. But I haven't implemented it yet.

There is a version of such fake client written in C. It's called chg. I 
know that at least sean farley have been playing with it.

>> Could we imagine an hybrid approach? where we stay in python as much as
>> possible and fallback to shell when needed?
>
> In theory. But there's all kinds of state you need to account for. You
> inevitably have to have something driving the test. You are either
> invoking a lot of shells or a lot of python processes. It becomes ugly
> fast. You really don't want to be make your test harness a source of
> bugs if at all possible.

I would image its is possible to stick all/most python either in the 
piloting process or in a simple child that runs for the whole test. Then 
falling back to shell when we have too. The amount of shell call should 
be limited. if we offer enough of the basic brick in python.

> Single mode tests or tests with clear
> delimiters for execution domains limit surface area for bugs.

I agree that simple is better. But the test suite is HUGE and a full 
rewrite does not seems realistic. An hybrid approach is the only way I 
see something usable widely in the near future.

-- 
Pierre-Yves