[Bug 4114] New: clone --uncompressed crash if source repo has file larger than 2GB and is on a 32-bits Linux host

Wed Nov 27 15:42:04 CST 2013

http://bz.selenic.com/show_bug.cgi?id=4114

          Priority: normal
            Bug ID: 4114
                CC: mercurial-devel at selenic.com
          Assignee: bugzilla at selenic.com
           Summary: clone --uncompressed crash if source repo has file
                    larger than 2GB and is on a 32-bits Linux host
          Severity: bug
    Classification: Unclassified
                OS: Linux
          Reporter: long at tlvu.ca
          Hardware: PC
            Status: UNCONFIRMED
           Version: 2.8
         Component: Mercurial
           Product: Mercurial

Notice the negative "-1857639037 bytes of data".

The error doesn't happen if the source repo is on a 64-bits host or if all its
files are smaller than 2G.

Mercurial 2.8, python 2.6.

[lvu at lvu:11 14:59:33 tmp]$ ~/src/mercurial-2.8/hg clone --uncompressed -U
http://forge/hg/intelepacs/master 
destination directory: master
streaming all changes
57035 files to transfer, -1857639037 bytes of data
** Python 2.6.6 (r266:84292, Jan 27 2011, 19:03:24) [GCC 4.4.4 20100630 (Red
Hat 4.4.4-10)]
** Mercurial Distributed SCM (version 2.8)
** Extensions loaded: graphlog, share, progress, ims.hg.clonerc, purge, rebase,
fetch, strip, mq, extdiff, ims.hg.goldilocks, ims.hg.backup, ims.hg.buildtags,
hgk, patchbomb, color, ims.hg.imshacks, churn, record, crecord
Traceback (most recent call last):
  File "/home/lvu/src/mercurial-2.8/hg", line 38, in <module>
    mercurial.dispatch.run()
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/dispatch.py", line 28, in
run
    sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255)
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/dispatch.py", line 69, in
dispatch
    ret = _runcatch(req)
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/dispatch.py", line 133, in
_runcatch
    return _dispatch(req)
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/dispatch.py", line 806, in
_dispatch
    cmdpats, cmdoptions)
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/dispatch.py", line 585, in
runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/extensions.py", line 196,
in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/oldhdd/home/lvu/src/mercurial-2.8/hgext/color.py", line 419, in
colorcmd
    return orig(ui_, opts, cmd, cmdfunc)
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/dispatch.py", line 897, in
_runcommand
    return checkargs()
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/dispatch.py", line 868, in
checkargs
    return cmdfunc()
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/dispatch.py", line 803, in
<lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/util.py", line 512, in
check
    return func(*args, **kwargs)
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/commands.py", line 1282,
in clone
    branch=opts.get('branch'))
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/extensions.py", line 196,
in wrap
    return wrapper(origfn, *args, **kwargs)
  File
"/usr/local/Intelerad/3rd_Party/python-2.6/lib/python2.6/site-packages/ims/hg/clonerc.py",
line 75, in hg_clone
    result = orig(ui, peeropts, source, dest=dest, **kwargs)
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/hg.py", line 372, in clone
    destpeer.local().clone(srcpeer, heads=revs, stream=stream)
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/localrepo.py", line 2425,
in clone
    return self.stream_in(remote, set(('revlogv1',)))
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/localrepo.py", line 2358,
in stream_in
    for chunk in util.filechunkiter(fp, limit=size):
  File "/oldhdd/home/lvu/src/mercurial-2.8/mercurial/util.py", line 993, in
filechunkiter
    assert limit is None or limit >= 0
AssertionError
[lvu at lvu:11 15:01:35 tmp]$ 

More investigation details below.

I appears the file size returned by repo.store.walk() is wrong (negative) if
the file is larger than 2G and the host is 32-bits Linux.

Since the file size is negative, that's why we have the AssertionError above.

A regular os.lstat('larger-than-2G-file').st_size on the same host returned the
proper value.

lvu at testforge:~/src/mercurial$ uname -a
Linux testforge 2.6.32-49-generic-pae #111-Ubuntu SMP Thu Jun 20 21:44:04 UTC
2013 i686 GNU/Linux

lvu at testforge:~/src/mercurial$ PYTHONPATH=~/src/mercurial python
Python 2.6.5 (r265:79063, Sep 26 2013, 18:51:09) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> from mercurial import __version__, hg, ui as hgui
>>> __version__.version
'2.8'
>>> ui = hgui.ui()
>>> repo = hg.repository(ui, "/src/hg/intelepacs/master-manifest-bigger-2G")
>>> os.lstat("/src/hg/intelepacs/master-manifest-bigger-2G/.hg/store/00manifest.d").st_size
2160494785L
>>> total_bytes = 0
>>> for name, ename, size in repo.store.walk():
...     if size:
...             total_bytes += size
...             if size < 0 or name == "00manifest.d":
...                     print "%s %s %d" % (name, ename, size)
... 
00manifest.d 00manifest.d -2134472511
>>> total_bytes
-1858194842L
>>> 
>>> repo = hg.repository(ui, "/src/hg/intelepacs/master-manifest-smaller-2G")
>>> os.lstat("/src/hg/intelepacs/master-manifest-smaller-2G/.hg/store/00manifest.d").st_size
1755040992L
>>> total_bytes = 0
>>> for name, ename, size in repo.store.walk():
...     if size:
...             total_bytes += size
...             if size < 0 or name == "00manifest.d":
...                     print "%s %s %d" % (name, ename, size)
... 
00manifest.d 00manifest.d 1755040992
>>> total_bytes
2021143148L
>>>

-- 
You are receiving this mail because:
You are on the CC list for the bug.