Mercurial bug cloning xen repo

Matt Mackall mpm at selenic.com
Tue Sep 20 13:18:07 CDT 2005


On Tue, Sep 20, 2005 at 10:35:34AM -0700, Matt Mackall wrote:
> On Tue, Sep 20, 2005 at 10:04:50AM -0700, Matt Mackall wrote:
> > On Tue, Sep 20, 2005 at 01:03:13PM +0100, Mike Wray wrote:
> > > I got the following error when trying to clone the xen unstable repo.
> > > This was using mercurial from a fresh pull today (see below for info)
> > > and python 2.4.1.
> > 
> > Hmm, I couldn't reproduce this.
> > 
> > > mjw at wray-m-3 1022> hg clone http://xenbits.xensource.com/xen-unstable.hg
> > > requesting all changes
> > > adding changesets
> > > adding manifests
> > > adding file changes
> > > added 6917 changesets with 0 changes to 0 files
> >                              ^^^^^^^^^^^^^^^^^^^^
> > Something bad happened there. I got:
> > 
> > requesting all changes
> > adding changesets
> > adding manifests
> > adding file changes
> > added 6964 changesets with 69703 changes to 7199 files
> 
> Ok, running on the theory that the http connection is closing
> prematurely, I did the following:
> 
> $ hg bundle ../xenbundle.hg ../empty
> searching for changes
> $ cd ..
> $ dd if=xenbundle.hg of=xentrunc.hg bs=1024 count=2000
> $ hg init xentrunc
> $ cd xentrunc/
> $ hg unbundle ../xentrunc.hg
> adding changesets
> adding manifests
> adding file changes
> added 6964 changesets with 0 changes to 0 files
>                            ^^^^^^^^^^^^^^^^^^^^
>  so we've truncated past the end of the changesets but before the
>  beginning of the file deltas
> 
> $ dd if=xenbundle.hg of=xentrunc.hg bs=1024 count=1024
> $ rm -rf xentrunc
> $ mkdir xentrunc; cd xentrunc
> $ hg init
> $ hg unbundle ../xentrunc.hg
> adding changesets
> adding manifests
> adding file changes
> added 6672 changesets with 0 changes to 0 files
>       ^^^^
> 
>  and now we've not got all the changesets.
> 
> The only difference between the bundle protocol and the wire protocol
> is the compression: the wire protocol uses zlib to reduce server load,
> while bundle uses bzip2. So I suspect what's happening is that the Xen
> webserver is cutting off connections that hit 1MB of data transfer in
> some situations. The fact that it's repeatable suggests a hard limit.

And then I added exception handling to the pull code, so that it now
does this:

abort: premature EOF reading chunk (got 75478 bytes, expected
1651992944)!

That expected number says there's probably something else in the
stream. Firing up Ethereal and tracing a connection shows:

<!-- The above is a description of an error in a Python program,
formatted
     for a Web browser because the 'cgitb' module was enabled.  In
     case you
     are not reading this in a Web browser, here is the original
     traceback:

Traceback (most recent call last):
  File "/var/www/html/xen-unstable.hg", line 9, in ?
    h.run()
  File "/usr/local/hg-0.6c/lib/python/mercurial/hgweb.py", line 701,
  in run
    chunk = f.read(4096)
  File "/usr/local/hg-0.6c/lib/python/mercurial/hg.py", line 1442, in
  read
    self.buf += self.g.next()
  File "/usr/local/hg-0.6c/lib/python/mercurial/hg.py", line 1469, in
  gengroup
    for y in self.manifest.group(linkmap): yield y
  File "/usr/local/hg-0.6c/lib/python/mercurial/revlog.py", line 433,
  in group
    chunks[r] = decompress(data[pos: pos + l])
  File "/usr/local/hg-0.6c/lib/python/mercurial/revlog.py", line 33,
  in decompress
    if t == 'x': return zlib.decompress(bin)
MemoryError

-->

So the server ran out of memory trying to decompress a manifest.

Unlike revlogs, which are only mostly delta-encoded, bundles and the
wire protocol are completely delta-encoded. Normally, this just reads
deltas that we've already calculated and sends them along, but when it
runs across full revisions, it re-encodes them.

Trouble is: none of these manifests are particularly big - even the
tip manifest is only 150k.

-- 
Mathematics is the supreme nostalgia of our time.


More information about the Mercurial mailing list