stable ordering of test output

[Pierre-Yves David]

On 03/07/2017 05:56 PM, Pierre-Yves David wrote:
>
>
> On 03/07/2017 05:49 PM, Augie Fackler wrote:
>> On Fri, Mar 03, 2017 at 04:37:54PM -0800, Jun Wu wrote:
>>> Excerpts from Danek Duvall's message of 2017-03-03 14:45:56 -0800:
>>>> I frequently get failures like this:
>>>>
>>>>     --- .../mercurial.hg/tests/test-bundle2-exchange.t
>>>>     +++ .../mercurial.hg/tests/test-bundle2-exchange.t.err
>>>>     @@ -1042,11 +1042,11 @@
>>>>        $ hg --config devel.legacy.exchange=bundle1 clone
>>>> ssh://user@dummy/bundle2onlyserver not-bundle2-ssh
>>>>        requesting all changes
>>>>        adding changesets
>>>>     -  remote: abort: incompatible Mercurial client; bundle2 required
>>>>     -  remote: (see
>>>> https://www.mercurial-scm.org/wiki/IncompatibleClient )
>>>>        transaction abort!
>>>>        rollback completed
>>>>        abort: stream ended unexpectedly (got 0 bytes, expected 4)
>>>>     +  remote: abort: incompatible Mercurial client; bundle2 required
>>>>     +  remote: (see
>>>> https://www.mercurial-scm.org/wiki/IncompatibleClient )
>>>>        [255]
>>>>
>>>>        $ cat > bundle2onlyserver/.hg/hgrc << EOF
>>>>
>>>>     ERROR: test-bundle2-exchange.t output changed
>>>>
>>>> It's usually fairly consistent, at least for a period of time, and
>>>> then it
>>>> goes away.  Presumably it's some sort of fairly stable timing issue,
>>>> and
>>>> possibly unique to the environment I'm running in (at least, I
>>>> assume that
>>>> the official tests aren't showing this).
>>>>
>>>> I could patch the tests locally to reorder the lines, but if it's
>>>> really an
>>>> environmental issue, then that's guaranteed to work consistently
>>>> even for
>>>> me.
>>>>
>>>> Is this (ever? frequently?) an issue for anyone else?
>>>
>>> Yes. We have seen this on our OSX tests. I guess it's "select()"
>>> returning
>>> different things.
>>
>> It's also a problem on the FreeBSD buildbot. I don't know enough about
>> the bundle2 code to understand how to fix it, but maybe we can figure
>> out a way to get marmoute an account on a machine taht would help him
>> diagnose? Danek, do you have a solaris machine you could get him
>> access to for testing purposes?
>
> Yeah, I tried to setup the BSD on the gcc compile farm to debug this,
> but the machine have such ancient everything else that I finally gave up
> (after recompiling my own python and a couple of dependency). I did not
> had time to spend on this since that last attempt. Having an account on
> something showing the issue would help.
>
> On the other hand, this is probably not so bundle2 specific. We have
> some "select" logic to read stdout and stderr as soon as possible. This
> is the main suspect as it is possible that this logic behave different
> under linux and other unix (not too much effort have been put into it).
> So there is not need of a deep knowledge of bundle2 to debug this if
> someone else want to give it a shot.

As perr Augie request on IRC let me be more specific: I would have a 
look at the behavior of the logic in sshpeer.py using "doublepipe" class 
and the associated "util.poll".

It is responsible for reading the first stream that has data (of stderr 
and stdout). This kind of change in output seems to imply that either 
the server is flushing the stream in different order or that the "ready 
to read" detection is disabled or behaving differently.

>>> The fix would be pretty straightforward while figuring out the root
>>> cause
>>> (what race condition it is) needs time.
>>>
>>>> I can't think of any particularly satisfying solution to this, other
>>>> than
>>>> perhaps separating the remote lines from the local lines, and comparing
>>>> each of those independently.  Would that make sense?
>>>>
>>>> Thanks,
>>>> Danek
>>> _______________________________________________
>>> Mercurial-devel mailing list
>>> Mercurial-devel at mercurial-scm.org
>>> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>

-- 
Pierre-Yves David