Mercurial bug cloning xen repo
Matt Mackall
mpm at selenic.com
Tue Sep 20 13:18:07 CDT 2005
On Tue, Sep 20, 2005 at 10:35:34AM -0700, Matt Mackall wrote:
> On Tue, Sep 20, 2005 at 10:04:50AM -0700, Matt Mackall wrote:
> > On Tue, Sep 20, 2005 at 01:03:13PM +0100, Mike Wray wrote:
> > > I got the following error when trying to clone the xen unstable repo.
> > > This was using mercurial from a fresh pull today (see below for info)
> > > and python 2.4.1.
> >
> > Hmm, I couldn't reproduce this.
> >
> > > mjw at wray-m-3 1022> hg clone http://xenbits.xensource.com/xen-unstable.hg
> > > requesting all changes
> > > adding changesets
> > > adding manifests
> > > adding file changes
> > > added 6917 changesets with 0 changes to 0 files
> > ^^^^^^^^^^^^^^^^^^^^
> > Something bad happened there. I got:
> >
> > requesting all changes
> > adding changesets
> > adding manifests
> > adding file changes
> > added 6964 changesets with 69703 changes to 7199 files
>
> Ok, running on the theory that the http connection is closing
> prematurely, I did the following:
>
> $ hg bundle ../xenbundle.hg ../empty
> searching for changes
> $ cd ..
> $ dd if=xenbundle.hg of=xentrunc.hg bs=1024 count=2000
> $ hg init xentrunc
> $ cd xentrunc/
> $ hg unbundle ../xentrunc.hg
> adding changesets
> adding manifests
> adding file changes
> added 6964 changesets with 0 changes to 0 files
> ^^^^^^^^^^^^^^^^^^^^
> so we've truncated past the end of the changesets but before the
> beginning of the file deltas
>
> $ dd if=xenbundle.hg of=xentrunc.hg bs=1024 count=1024
> $ rm -rf xentrunc
> $ mkdir xentrunc; cd xentrunc
> $ hg init
> $ hg unbundle ../xentrunc.hg
> adding changesets
> adding manifests
> adding file changes
> added 6672 changesets with 0 changes to 0 files
> ^^^^
>
> and now we've not got all the changesets.
>
> The only difference between the bundle protocol and the wire protocol
> is the compression: the wire protocol uses zlib to reduce server load,
> while bundle uses bzip2. So I suspect what's happening is that the Xen
> webserver is cutting off connections that hit 1MB of data transfer in
> some situations. The fact that it's repeatable suggests a hard limit.
And then I added exception handling to the pull code, so that it now
does this:
abort: premature EOF reading chunk (got 75478 bytes, expected
1651992944)!
That expected number says there's probably something else in the
stream. Firing up Ethereal and tracing a connection shows:
<!-- The above is a description of an error in a Python program,
formatted
for a Web browser because the 'cgitb' module was enabled. In
case you
are not reading this in a Web browser, here is the original
traceback:
Traceback (most recent call last):
File "/var/www/html/xen-unstable.hg", line 9, in ?
h.run()
File "/usr/local/hg-0.6c/lib/python/mercurial/hgweb.py", line 701,
in run
chunk = f.read(4096)
File "/usr/local/hg-0.6c/lib/python/mercurial/hg.py", line 1442, in
read
self.buf += self.g.next()
File "/usr/local/hg-0.6c/lib/python/mercurial/hg.py", line 1469, in
gengroup
for y in self.manifest.group(linkmap): yield y
File "/usr/local/hg-0.6c/lib/python/mercurial/revlog.py", line 433,
in group
chunks[r] = decompress(data[pos: pos + l])
File "/usr/local/hg-0.6c/lib/python/mercurial/revlog.py", line 33,
in decompress
if t == 'x': return zlib.decompress(bin)
MemoryError
-->
So the server ran out of memory trying to decompress a manifest.
Unlike revlogs, which are only mostly delta-encoded, bundles and the
wire protocol are completely delta-encoded. Normally, this just reads
deltas that we've already calculated and sends them along, but when it
runs across full revisions, it re-encodes them.
Trouble is: none of these manifests are particularly big - even the
tip manifest is only 150k.
--
Mathematics is the supreme nostalgia of our time.
More information about the Mercurial
mailing list