MQ performance on large repo
Greg Ward
greg at gerg.ca
Thu Mar 4 08:45:22 CST 2010
On Thu, Mar 4, 2010 at 3:50 AM, Benoit Boissinot <bboissin at gmail.com> wrote:
> What if you disable lazyindex? The C parser might be faster anyway.
What, you mean like this:
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -369,7 +369,7 @@
def parseindex(self, fp, data, inline):
if len(data) == _prereadsize:
- if util.openhardlinks() and not inline:
+ if False and util.openhardlinks() and not inline:
# big index, let's parse it on demand
parser = lazyparser(fp)
index = lazyindex(parser)
? OK, I'll give it a try. Also I've disabled local hooks and
extensions, which shaves about a second off of qrefresh. So here's a
new profile run with unpatched revlog:
"""
CallCount Recursive Total(ms) Inline(ms) module:lineno(function)
3 0 2.1548 0.7854 mercurial.revlog:1313(strip)
+213165 0 0.7978 0.4946
+mercurial.revlog:269(__getitem__)
+213187 0 0.1661 0.1660
+mercurial.revlog:504(__iter__)
+3 0 0.0000 0.0000 +mercurial.revlog:522(start)
+5 0 0.0001 0.0000
+mercurial.transaction:19(_active)
+2 0 0.4052 0.0000
+mercurial.revlog:487(_loadindexmap)
320633 0 1.4827 0.7373
mercurial.revlog:269(__getitem__)
+320633 0 0.4560 0.4560 +<_struct.unpack>
+115 0 0.2894 0.0004 +mercurial.revlog:264(load)
269 0 0.6404 0.6404 <zlib.decompress>
336 0 0.6928 0.6177
mercurial.revlog:164(loadblock)
+336 0 0.0734 0.0734 +<method 'read' of
'file' objects>
+336 0 0.0012 0.0012 +<method 'seek' of
'file' objects>
+672 0 0.0005 0.0005 +<len>
+8 0 0.0000 0.0000 +<max>
320643 0 0.4561 0.4561 <_struct.unpack>
2 0 0.3948 0.3835
mercurial.revlog:137(loadmap)
+834 0 0.0113 0.0113 +<method 'read' of
'file' objects>
+2 0 0.0000 0.0000 +<method 'seek' of
'file' objects>
106601 0 0.9872 0.3235
mercurial.revlog:514(linkrev)
+106582 0 0.6637 0.2399
+mercurial.revlog:269(__getitem__)
19235 0 0.2601 0.2601 <posix.lstat>
3 0 1.2804 0.2488 mercurial.dirstate:433(walk)
+2705 0 0.1908 0.1908 +<mercurial.osutil.listdir>
+2 0 0.3574 0.0452 +<zip>
+5 0 0.0315 0.0315 +<sorted>
+37240 0 0.0262 0.0262
+mercurial.match:74(<lambda>)
+21901 0 0.0250 0.0250
+mercurial.dirstate:126(_join)
2 0 1.2040 0.2167
mercurial.repair:35(collectone)
+106601 0 0.9872 0.3235
+mercurial.revlog:514(linkrev)
+2 0 0.0000 0.0000 +<len>
Time: real 7.770 secs (user 6.130+0.000 sys 1.580+0.000)
"""
and here's the same result after sabotaging revlogio.parseindex():
"""
CallCount Recursive Total(ms) Inline(ms) module:lineno(function)
10 0 1.1327 1.1327
<mercurial.parsers.parse_index>
269 0 0.6468 0.6468 <zlib.decompress>
19235 0 0.2606 0.2606 <posix.lstat>
3 0 0.4049 0.2596 mercurial.revlog:1313(strip)
+213187 0 0.1451 0.1451
+mercurial.revlog:504(__iter__)
+5 0 0.0001 0.0000
+mercurial.transaction:19(_active)
+3 0 0.0000 0.0000 +mercurial.revlog:522(start)
+6 0 0.0000 0.0000 +<len>
+3 0 0.0000 0.0000 +<isinstance>
3 0 1.2805 0.2459 mercurial.dirstate:433(walk)
+2705 0 0.1896 0.1896 +<mercurial.osutil.listdir>
+2 0 0.3628 0.0499 +<zip>
+5 0 0.0314 0.0314 +<sorted>
+37240 0 0.0263 0.0263
+mercurial.match:74(<lambda>)
+21901 0 0.0258 0.0258
+mercurial.dirstate:126(_join)
2721 0 0.1924 0.1924 <built-in method match>
2705 0 0.1896 0.1896 <mercurial.osutil.listdir>
4 0 0.1888 0.1888 <delattr>
2 0 0.2511 0.1524
mercurial.repair:35(collectone)
+106601 0 0.0987 0.0987
+mercurial.revlog:514(linkrev)
+2 0 0.0000 0.0000 +<len>
28570 0 0.2527 0.1455
mercurial.store:24(decodedir)
+85710 0 0.0786 0.0786 +<method 'replace'
of 'str' objects>
+28570 0 0.0287 0.0287 +<method
'startswith' of 'str' objects>
Time: real 5.920 secs (user 4.570+0.000 sys 1.350+0.000)
"""
I ran each one a couple of times, and the speedup from ~7.7 sec to
~5.9 sec was consistent. Hmmmm! (Those are runtimes with profiling
overhead, of course. With hooks and one local extension disabled,
revlogio sabotaged, and profiling turned off, qrefresh is now around
4.4 sec.)
Greg
More information about the Mercurial-devel
mailing list