stable ordering of test output

[Danek Duvall]

Pierre-Yves David wrote:

> >>It's also a problem on the FreeBSD buildbot. I don't know enough about
> >>the bundle2 code to understand how to fix it, but maybe we can figure
> >>out a way to get marmoute an account on a machine taht would help him
> >>diagnose? Danek, do you have a solaris machine you could get him
> >>access to for testing purposes?
> >
> >Yeah, I tried to setup the BSD on the gcc compile farm to debug this,
> >but the machine have such ancient everything else that I finally gave up
> >(after recompiling my own python and a couple of dependency). I did not
> >had time to spend on this since that last attempt. Having an account on
> >something showing the issue would help.
> >
> >On the other hand, this is probably not so bundle2 specific. We have
> >some "select" logic to read stdout and stderr as soon as possible. This
> >is the main suspect as it is possible that this logic behave different
> >under linux and other unix (not too much effort have been put into it).
> >So there is not need of a deep knowledge of bundle2 to debug this if
> >someone else want to give it a shot.
> 
> As perr Augie request on IRC let me be more specific: I would have a look at
> the behavior of the logic in sshpeer.py using "doublepipe" class and the
> associated "util.poll".
> 
> It is responsible for reading the first stream that has data (of stderr and
> stdout). This kind of change in output seems to imply that either the server
> is flushing the stream in different order or that the "ready to read"
> detection is disabled or behaving differently.

It is in fact not specific to bundle2, as I'm also currently seeing the
problem in test-ssh-bundle1.

I would think this is an inherently racy problem, as with any asynchronous
execution, though I would then expect more variance in the results, so
perhaps I'm mistaken here.

It looks like my home machine reproduces the problem.  I'll set you up with
an account and send you the login info.

Thanks,
Danek