stable ordering of test output

[Augie Fackler]

On Wed, Mar 8, 2017 at 10:44 AM, Yuya Nishihara <yuya at tcha.org> wrote:
> On Tue, 7 Mar 2017 17:56:58 +0100, Pierre-Yves David wrote:
>> On the other hand, this is probably not so bundle2 specific. We have
>> some "select" logic to read stdout and stderr as soon as possible. This
>> is the main suspect as it is possible that this logic behave different
>> under linux and other unix (not too much effort have been put into it).
>
> posix.poll() waits every type of operation no matter if fd is e.g. writable
> or not. IIRC, this doesn't always work on FreeBSD since the underlying resource
> of read/write ends might be shared in the kernel.
>
> But I don't think this is the source of the unstable output.

I've had a little time today between things to try and debug this.
What I've found so far:

1) when the test passes, the remote: output is printed by the
_forwardoutput method in sshpeer, presumably since stderr makes it to
the client before the close of stdout.
2) When the test fails (as on BSD, and I guess Solaris), the client
notices that stdout closed before stderr. It then aborts the
transaction and sshpeer.cleanup() notices some data chilling on stderr
and ensures it gets read and printed.

I'm not really sure why BSD systems would be quicker at communicating
the closed FD than other systems. I'm poking at dummyssh now to see if
maybe it's weirdness from there...