A possible explanation for random "stream ended unexpectedly (got m bytes, expected n)"

Mike Hommey mh at glandium.org
Sat Mar 25 11:19:07 UTC 2017


Hi,

I don't know about you, but occasionally, I've hit "stream ended
unexpectedly (got m bytes, expected n)" errors that didn't make sense.
Retrying would always work.

Recently, I was trying to use signal.setitimer and a signal handler for
some memory profiling on git-cinnabar, which uses mercurial as a
library, and got "stream ended 4 unexpectedly (got m bytes, expected n)"
*very* reproducibly. Like, with an interval timer firing every second,
it took only a few seconds to hit the error during a clone.

I'm pretty sure this can be reproduced with a similar setup in mercurial
itself.

Now, the reason this happens in this case is that, the code that fails
does:

def readexactly(stream, n):
    '''read n bytes from stream.read and abort if less was available'''
    s = stream.read(n)
    if len(s) < n:
        raise error.Abort(_("stream ended unexpectedly"
                           " (got %d bytes, expected %d)")
                          % (len(s), n))
    return s

... and thanks to POSIX, interrupted reads can lead to short reads. So,
you request n bytes, and get less, just because something interrupted
the process.  The problem then is that python doesn't let you know why
you just got a short read, and you have to figure that out on your own.

The same kind of problem is also true to some extent on writes.

Now, the problem is that this piece of code is likely the most visible
place where the issue exists, but there are many other places in the
mercurial code base that are likely affected.

And while the signal.setitimer case is a corner case (and I ended up
using a separate thread to work around the problem ; my code wasn't
interruption safe either anyways), I wonder if those random "stream
ended unexpectedly (got m bytes, expected n)" errors I was getting under
normal circumstances are not just a manifestation of the same underlying
issue, which is that the code doesn't like interrupted reads.

Disclaimer: I'm not going to work on fixing this issue, but I figured
I'd let you know, in case someone wants to look into it more deeply.

Cheers,

Mike


More information about the Mercurial-devel mailing list