Got this when tried to update to a changeset with a particularly large file changed (about 1.2 GB): ** unknown exception encountered, please report by visiting ** http://mercurial.selenic.com/wiki/BugTracker ** Python 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit (AMD64)] ** Mercurial Distributed SCM (version 2.9.2) ** Extensions loaded: Traceback (most recent call last): File "hg", line 42, in <module> File "mercurial\dispatch.pyo", line 28, in run File "mercurial\dispatch.pyo", line 69, in dispatch File "mercurial\dispatch.pyo", line 138, in _runcatch File "mercurial\dispatch.pyo", line 810, in _dispatch File "mercurial\dispatch.pyo", line 590, in runcommand File "mercurial\dispatch.pyo", line 901, in _runcommand File "mercurial\dispatch.pyo", line 872, in checkargs File "mercurial\dispatch.pyo", line 807, in <lambda> File "mercurial\util.pyo", line 511, in check File "mercurial\commands.pyo", line 5889, in update File "mercurial\hg.pyo", line 472, in update File "mercurial\hg.pyo", line 468, in updaterepo File "mercurial\merge.pyo", line 927, in update File "mercurial\merge.pyo", line 623, in applyupdates File "mercurial\merge.pyo", line 544, in getremove File "mercurial\context.pyo", line 804, in data File "mercurial\filelog.pyo", line 38, in read File "mercurial\revlog.pyo", line 998, in revision File "mercurial\revlog.pyo", line 912, in _chunks File "mercurial\revlog.pyo", line 891, in _chunkraw File "mercurial\revlog.pyo", line 882, in _getchunk File "mercurial\revlog.pyo", line 863, in _loadchunk OverflowError: Python int too large to convert to C long
Is the repository public?
It is not... :(
Can you send the output of: hg debugindex YOURFILE hg debugrevlog YOURFILE ..as well as the revision numbers you're trying to move between? My guess of what's going on here: - your final object is 1.2GB - reconstructing it can take up 2.4GB of deltas - Mercurial attempts to read all that data at once - Python's read method is expecting an argument it can convert to a C 'long' - on Windows 64 (but not on basically every other 64-bit system), a long is still only 32 bits - 2.4G causes a signed overflow https://en.wikipedia.org/wiki/64-bit_computing#64-bit_data_models
I'm trying to move between revs 4055 (eabe8d378a16) and 4056 (b94ffb2790aa). hg debugindex: rev offset length base linkrev nodeid p1 p2 0 0 513067849 0 1950 cfbf52bf0cdb 000000000000 000000000000 1 513067849 485214889 0 2012 dd6b8bedb32b cfbf52bf0cdb 000000000000 2 998282738 376664590 2 2389 59022c0ccf96 dd6b8bedb32b 000000000000 3 1374947328 413858561 2 2410 bdf27a8c964d 59022c0ccf96 000000000000 4 1788805889 248034396 4 2534 eb2ab0ba6b9c bdf27a8c964d 000000000000 5 2036840285 414142691 4 2613 769e7cd81fd9 eb2ab0ba6b9c 000000000000 6 2450982976 592286230 6 3350 8b1c9431aecb 769e7cd81fd9 000000000000 7 3043269206 594460658 6 3431 99fd1ff10977 8b1c9431aecb 000000000000 8 3637729864 496043003 8 3778 18330b70510b 99fd1ff10977 000000000000 9 4133772867 633413616 8 3827 91b494a06597 18330b70510b 000000000000 10 4767186483 1107823421 10 3907 9594fc174e30 91b494a06597 000000000000 11 5875009904 1116915161 10 4056 07935d253045 9594fc174e30 000000000000 hg debugrevlog: format : 1 flags : (none) revisions : 12 merges : 0 ( 0.00%) normal : 12 (100.00%) revisions : 12 full : 6 (50.00%) deltas : 6 (50.00%) revision size : 6991925065 full : 3333919489 (47.68%) deltas : 3658005576 (52.32%) avg chain length : 0 compression ratio : 1 uncompressed data size (min/max/avg) : 248034395 / 1116915053 / 585166488 full revision size (min/max/avg) : 248034396 / 1107823421 / 555653248 delta size (min/max/avg) : 413858561 / 1116915161 / 609667596 deltas against prev : 6 (100.00%) where prev = p1 : 6 (100.00%) where prev = p2 : 0 ( 0.00%) other : 0 ( 0.00%)
(In reply to comment #3) I thought Mercurial don't use deltas if they result in being bigger than the object verbatim?
The bound is 2x the object size. Looks like reconstructing the last revision is going to require a read of 1107823421 + 1116915161 bytes, about 77MB past the 2G boundary. (There's no corresponding problem with write or seek, as Python correctly uses a 64-bit ssize_t here.)
Possible fix: diff -r dbf0fa39a5b8 mercurial/revlog.py --- a/mercurial/revlog.py Fri Apr 04 16:41:51 2014 -0700 +++ b/mercurial/revlog.py Mon Apr 07 14:13:52 2014 -0500 @@ -913,8 +913,13 @@ ladd = l.append # preload the cache - self._chunkraw(revs[0], revs[-1]) - offset, data = self._chunkcache + try: + self._chunkraw(revs[0], revs[-1]) + offset, data = self._chunkcache + except OverflowError: + # issue4215 - we can't cache a run of chunks greater than + # 2G on Windows + return [self._chunk(rev) for rev in revs] for rev in revs: chunkstart = start(rev) Please test if possible.
(In reply to comment #7) With that change my pure Python Mercurial crashed with out of memory exception... :(
Running 64-bit Python on 64-bit Windows? How much RAM?
(In reply to comment #9) Yes, 64-bit OS and 64-bit Python, 8 GB of RAM. I tried to run it again, and this time my workstation just froze with zero responsiveness (had to reboot after 15 minutes of waiting anything to happen)
I wonder if Mercurial could internally split any "large" files into smaller and more manageable chunks (of let's say 256MB) and only do deltas within boundaries of those chunks?..
Fixed by http://selenic.com/repo/hg/rev/469d949a7cb8 Matt Mackall <mpm@selenic.com> revlog: deal with chunk ranges over 2G on Windows (issue4215) Python uses a C long (32 bits on Windows 64) rather than an ssize_t in read(), and thus has a 2G size limit. Work around this by falling back to reading one chunk at a time on overflow. This approximately doubles our headroom until we run back into the size limit on single reads. (please test the fix)
Python upstream bug: http://bugs.python.org/issue21199
Do you have any form of swap enabled?
(In reply to comment #14) Yes, I do have normal Windows swap enabled as well 30GB of flash drive dedicated for ReadyBoost.
Do you have access to a Linux machine where you can attempt a checkout? If so, if you can capture the output of 'vmstat 1' while the update is running, we can get a sense of the memory use.
(In reply to comment #16) I did a few more tries on Windows: > hg update -C -r 4056 abort: out of memory > hg update -C -r 4055 0 files updated, 0 files merged, 0 files removed, 0 files unresolved > hg update -C -r 4056 1 files updated, 0 files merged, 0 files removed, 0 files unresolved So it managed to work at the end but I've seen Python taking about 3.3 of RAM before system went unresponsive.
(In reply to comment #17) (I meant 3.3 GB of RAM)