A possible explanation for random "stream ended unexpectedly (got m bytes, expected n)"

[Sean Farley]

Matt Harbison <mharbison72 at gmail.com> writes:

> On Tue, 06 Jun 2017 22:30:23 -0400, Matt Harbison <mharbison72 at gmail.com>  
> wrote:
>>
>> Any more info on this?
>>
>> We are starting to see this for a handful of developers on various  
>> repos.  So far, every instance has been cloning on Windows 7, from a  
>> CentOS 7 server.  The strange aspect is that it is *very* consistent for  
>> the people it strikes.  One developer had it happen every time over a  
>> period of days with both http and https.  But then when he tried to get  
>> a Wireshark trace, the problem stopped.
>>
>> Today, a new guy on a fresh install of Windows 7 + updates + Visual  
>> Studio got the "0 bytes, 4 expected" message on a very small repo (less  
>> than 50 commits I bet).  I upgraded him to 4.2.1 because I remembered  
>> there was work in this area to not swallow the actual error, and it  
>> coughed up these hairballs:
>>
>> d:\dev\sw>hg clonevcs sw/toolchain/tpos --traceback
>> destination directory: tpos
>> requesting all changes
>> adding changesets
>> adding manifests
>> adding file changes
>> transaction abort!
>> rollback completed
>> Traceback (most recent call last):
>>    File "mercurial\scmutil.pyo", line 146, in callcatch
>>    File "mercurial\dispatch.pyo", line 285, in _runcatchfunc
>>    File "mercurial\dispatch.pyo", line 912, in _dispatch
>>    File "mercurial\dispatch.pyo", line 648, in runcommand
>>    File "mercurial\dispatch.pyo", line 920, in _runcommand
>>    File "mercurial\dispatch.pyo", line 909, in <lambda>
>>    File "mercurial\util.pyo", line 1080, in check
>>    File "mercurial\dispatch.pyo", line 506, in __call__
>>    File "mercurial\util.pyo", line 1080, in check
>>    File "mercurial\commands.pyo", line 1561, in clone
>>    File "mercurial\hg.pyo", line 619, in clone
>>    File "mercurial\exchange.pyo", line 1238, in pull
>>    File "mercurial\exchange.pyo", line 1378, in _pullbundle2
>>    File "mercurial\bundle2.pyo", line 387, in processbundle
>> SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1752)
>> abort: error: WRONG_VERSION_NUMBER
>>
>> d:\dev\sw>hg clonevcs sw/toolchain/tpos --traceback --config  
>> extensions.mercurial_keyring=!
>> http authorization required for
>> https://vcs.attotech.com/hg/sw/toolchain/tpos
>> realm: SONIA :: SCM Manager
>> user: ssiwinski
>> password:
>> destination directory: tpos
>> requesting all changes
>> adding changesets
>> adding manifests
>> adding file changes
>> transaction abort!
>> rollback completed
>> Traceback (most recent call last):
>>    File "mercurial\scmutil.pyo", line 146, in callcatch
>>    File "mercurial\dispatch.pyo", line 285, in _runcatchfunc
>>    File "mercurial\dispatch.pyo", line 912, in _dispatch
>>    File "mercurial\dispatch.pyo", line 648, in runcommand
>>    File "mercurial\dispatch.pyo", line 920, in _runcommand
>>    File "mercurial\dispatch.pyo", line 909, in <lambda>
>>    File "mercurial\util.pyo", line 1080, in check
>>    File "mercurial\dispatch.pyo", line 506, in __call__
>>    File "mercurial\util.pyo", line 1080, in check
>>    File "mercurial\commands.pyo", line 1561, in clone
>>    File "mercurial\hg.pyo", line 619, in clone
>>    File "mercurial\exchange.pyo", line 1238, in pull
>>    File "mercurial\exchange.pyo", line 1378, in _pullbundle2
>>    File "mercurial\bundle2.pyo", line 387, in processbundle
>> SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed  
>> or bad record mac (_ssl.c:1752)
>> abort: error: DECRYPTION_FAILED_OR_BAD_RECORD_MAC
>>
>> For him, it worked fine over http.  (The schemes extension is hiding the  
>> https:// here.)
>>
>> The thing I'm going to try tomorrow is to do `hg serve` with the  
>> certificate like the test suite, to try to eliminate tomcat and  
>> scmmanager as suspects.  I forgot about the generaldelta tidbit, so I'll  
>> have to check that tomorrow too.  The repos were created with 3.9 IIRC,  
>> and everybody should be on 4.0.2 locally.  (Is there a debug command for  
>> that with 3.9?)
>
> Today, we got it to fail over https with a simple `hg serve` from the same  
> server (the output there indicated a broken pipe in  
> self._sslobj.write(data) ssl.py:689).  The server side has a 3.9 install,  
> and the repo is not generaldelta.  The server side output looked like it  
> made it further in one attempt than the other.
>
> It also failed over http, served from my Linux development machine and two  
> of my Windows 7 machines.  The client side indicated zlib errors.  I'm  
> starting to wonder if this is a hardware problem of some sort.  That  
> doesn't seem like a satisfying answer though, because apparently they were  
> remoted into this same client machine for several hours without an issue,  
> and could also clone a much larger repo.  I would think if the problem is  
> bad enough to fail every time on this repo, other things would be affected  
> too.

I haven't tried this with 'hg serve' but I must note that we've seen
this with windows and also run CentOS 7 on our servers. Not a smoking
gun, of course.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 800 bytes
Desc: not available
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20170611/d7041949/attachment.sig>