Speed on Windows for big repos
Adrian Buehlmann
adrian at cadifra.com
Mon Jan 28 13:01:31 CST 2008
On 28.01.2008 01:56, Matt Mackall wrote:
> On Sun, 2008-01-27 at 18:52 -0600, Matt Mackall wrote:
>> On Mon, 2008-01-28 at 01:09 +0100, Adrian Buehlmann wrote:
>>> > hg clone --noupdate --time --lsprof http://www.selenic.com/hg/ mercurial
>>> requesting all changes
>>> adding changesets
>>> adding manifests
>>> adding file changes
>>> added 5930 changesets with 11128 changes to 800 files
>>> CallCount Total(s) Inline(s) module:lineno(function)
>>> 3859 105.8043 105.8043 <win32file.FlushFileBuffers>
> ...
>> So what do we need to do for flush() here? Probably nothing at all. As
>> far as I can tell, WriteFile() is more or less analogous to write(2) in
>> UNIX, which doesn't have any application-side buffering so flush is
>> unneeded.
>
> And here's a patch:
>
> diff -r 359036c4e09e mercurial/util_win32.py
> --- a/mercurial/util_win32.py Fri Jan 25 17:10:35 2008 -0600
> +++ b/mercurial/util_win32.py Sun Jan 27 18:56:12 2008 -0600
> @@ -317,10 +317,7 @@
> self.closed = True
>
> def flush(self):
> - try:
> - win32file.FlushFileBuffers(self.handle)
> - except pywintypes.error, err:
> - raise WinIOError(err)
> + pass
>
> def truncate(self, pos=0):
> try:
>
Much faster now (patch applied to 0.9.5):
> hg clone --time --noupdate --pull netbeans-main netbeans-clone
requesting all changes
adding changesets
adding manifests
adding file changes
added 64894 changesets with 325167 changes to 74741 files
Time: real 984.002 secs (user 327.484+0.000 sys 455.719+0.000)
Without the patch:
> hg clone --time --noupdate --pull netbeans-main netbeans-clone
requesting all changes
adding changesets
adding manifests
adding file changes
added 64894 changesets with 325167 changes to 74741 files
Time: real 4151.165 secs (user 318.266+0.000 sys 440.734+0.000)
With the patch, profiled:
> hg clone --time --noupdate --pull --lsprof netbeans-main netbeans-clone
requesting all changes
adding changesets
adding manifests
adding file changes
added 64894 changesets with 325167 changes to 74741 files
CallCount Total(s) Inline(s) module:lineno(function)
467836 239.0337 239.0337 <win32file.ReadFile>
497427 167.5576 167.5576 <zlib.compress>
474195 98.8893 98.8893 <win32file.CreateFile>
6365852 52.4442 40.6655 mercurial.demandimport:71(__getattribute__)
+3182900 12.7929 12.7929 +mercurial.demandimport:71(__getattribute__)
+1591450 13.6548 7.7736 +mercurial.demandimport:43(_load)
+1591450 3.7622 3.7622 +<getattr>
74743 1146.6750 38.0099
<c:\python25\lib\site-packages\mercurial\revlog.py>:1124(addgroup)
+1049877 31.9662 13.0174 +mercurial.util_win32:258(write)
+104996 120.2166 9.7463
+<c:\python25\lib\site-packages\mercurial\revlog.py>:1014(_addrevision)
+380212 114.5439 4.0905
+<c:\python25\lib\site-packages\mercurial\revlog.py>:60(compress)
+454955 6.1762 3.2686
+<c:\python25\lib\site-packages\mercurial\revlog.py>:531(end)
+74743 5.4181 2.8696
+<c:\python25\lib\site-packages\mercurial\transaction.py>:42(add)
190342 30.6219 30.6219 <nt.stat>
1134136 659.0645 25.7153
<c:\python25\lib\site-packages\mercurial\util.py>:1406(read)
+1589505 615.8859 12.5750
+<c:\python25\lib\site-packages\mercurial\localrepo.py>:1780(gengroup)
+1600204 8.2699 8.2699 +<method 'write' of 'cStringIO.StringO'
objects>
+3868476 6.0949 6.0949 +<len>
+10700 2.4900 2.4900 +<method 'getvalue' of
'cStringIO.StringO' objects>
+10700 0.5144 0.2033 +mercurial.demandimport:71(__getattribute__)
1049396 25.5865 25.5865 <win32file.WriteFile>
633769 25.5201 25.5201 <zlib.decompress>
13622183 21.2767 21.2767 <len>
+4 0.0000 0.0000
+<c:\python25\lib\site-packages\mercurial\revlog.py>:249(__len__)
Time: real 1264.443 secs (user 511.281+0.000 sys 586.063+0.000)
> hg clone --noupdate --time --lsprof http://www.selenic.com/hg/ mercurial
requesting all changes
adding changesets
adding manifests
adding file changes
added 5930 changesets with 11128 changes to 800 files
CallCount Total(s) Inline(s) module:lineno(function)
4888 71.1324 71.1324 <method 'recv' of '_socket.socket' objects>
27104 5.4454 5.4454 <zlib.compress>
802 87.8195 1.6812
<c:\python25\lib\site-packages\mercurial\revlog.py>:1124(addgroup)
+57681 1.6066 0.6775 +mercurial.util_win32:258(write)
+3759 2.8091 0.3360
+<c:\python25\lib\site-packages\mercurial\revlog.py>:1014(_addrevision)
+22184 4.4088 0.2093
+<c:\python25\lib\site-packages\mercurial\revlog.py>:60(compress)
+3759 2.8602 0.1836
+<c:\python25\lib\site-packages\mercurial\revlog.py>:915(revision)
+22986 0.2923 0.1532
+<c:\python25\lib\site-packages\mercurial\revlog.py>:531(end)
83772 2.3385 0.9642 mercurial.util_win32:258(write)
+60246 0.6583 0.6583 +<win32file.WriteFile>
+68909 0.5087 0.5087 +<win32file.SetFilePointer>
+144018 0.2073 0.2073 +<len>
7418 0.7722 0.7722 <win32file.CreateFile>
439336 0.6599 0.6599 <len>
+10 0.0000 0.0000 +sre_parse:132(__len__)
60246 0.6583 0.6583 <win32file.WriteFile>
89646 0.5980 0.5980 <win32file.SetFilePointer>
48375 71.0479 0.5167
<c:\python25\lib\site-packages\mercurial\util.py>:1406(read)
+96872 0.1519 0.1519 +<len>
+61 0.0360 0.0360 +<method 'getvalue' of
'cStringIO.StringO' objects>
+122 0.0359 0.0359 +<method 'write' of 'cStringIO.StringO'
objects>
+61 70.3038 0.0019
+<c:\python25\lib\site-packages\mercurial\httprepo.py>:138(zgenerator)
+61 0.0031 0.0011 +mercurial.demandimport:71(__getattribute__)
12037 2.3964 0.4818
<c:\python25\lib\site-packages\mercurial\revlog.py>:865(chunk)
+2570 1.4696 0.1052
+<c:\python25\lib\site-packages\mercurial\revlog.py>:866(loadcache)
+12037 0.0533 0.0533
+<c:\python25\lib\site-packages\mercurial\revlog.py>:529(start)
+12037 0.3395 0.0521
+<c:\python25\lib\site-packages\mercurial\revlog.py>:89(decompress)
+12037 0.0253 0.0253
+<c:\python25\lib\site-packages\mercurial\revlog.py>:533(length)
+11964 0.0173 0.0173 +<len>
Time: real 91.909 secs (user 7.766+0.000 sys 12.547+0.000)
> hg clone --pull --noupdate --time --lsprof mercurial mercurial-cloned
requesting all changes
adding changesets
adding manifests
adding file changes
added 5930 changesets with 11128 changes to 800 files
CallCount Total(s) Inline(s) module:lineno(function)
27104 6.3940 6.3940 <zlib.compress>
15087 2.3581 2.3581 <win32file.ReadFile>
802 32.1878 1.7014
<c:\python25\lib\site-packages\mercurial\revlog.py>:1124(addgroup)
+57681 1.6272 0.6930 +mercurial.util_win32:258(write)
+3759 3.0678 0.3597
+<c:\python25\lib\site-packages\mercurial\revlog.py>:1014(_addrevision)
+22184 5.1158 0.2135
+<c:\python25\lib\site-packages\mercurial\revlog.py>:60(compress)
+3759 2.8403 0.1838
+<c:\python25\lib\site-packages\mercurial\revlog.py>:915(revision)
+22986 0.3109 0.1656
+<c:\python25\lib\site-packages\mercurial\revlog.py>:531(end)
19930 1.3291 1.3291 <win32file.CreateFile>
55787 4.6788 1.2388
<c:\python25\lib\site-packages\mercurial\revlog.py>:865(chunk)
+55787 0.2446 0.2446
+<c:\python25\lib\site-packages\mercurial\revlog.py>:529(start)
+55787 1.4151 0.2391
+<c:\python25\lib\site-packages\mercurial\revlog.py>:89(decompress)
+55787 0.1175 0.1175
+<c:\python25\lib\site-packages\mercurial\revlog.py>:533(length)
+2709 1.5696 0.1082
+<c:\python25\lib\site-packages\mercurial\revlog.py>:866(loadcache)
+55709 0.0838 0.0838 +<len>
42454 1.1759 1.1759 <zlib.decompress>
48375 14.4406 0.9837
<c:\python25\lib\site-packages\mercurial\util.py>:1406(read)
+71362 12.9395 0.3187
+<c:\python25\lib\site-packages\mercurial\localrepo.py>:1780(gengroup)
+168323 0.2537 0.2537 +<len>
+71573 0.2208 0.2208 +<method 'write' of 'cStringIO.StringO'
objects>
+212 0.0311 0.0311 +<method 'getvalue' of
'cStringIO.StringO' objects>
+212 0.0101 0.0039 +mercurial.demandimport:71(__getattribute__)
83772 2.3723 0.9836 mercurial.util_win32:258(write)
+60246 0.6857 0.6857 +<win32file.WriteFile>
+68909 0.4920 0.4920 +<win32file.SetFilePointer>
+144018 0.2110 0.2110 +<len>
644651 0.9737 0.9737 <len>
19362 7.5911 0.8036
<c:\python25\lib\site-packages\mercurial\revlog.py>:915(revision)
+36560 3.7254 0.9072
+<c:\python25\lib\site-packages\mercurial\revlog.py>:865(chunk)
+15196 0.5757 0.2315
+<c:\python25\lib\site-packages\mercurial\revlog.py>:46(hash)
+15196 0.2687 0.1618
+<c:\python25\lib\site-packages\mercurial\revlog.py>:524(parents)
+15196 0.1439 0.1439 +<mercurial.mpatch.patches>
+11712 1.8780 0.1183
+<c:\python25\lib\site-packages\mercurial\util.py>:1259(o)
Time: real 35.117 secs (user 15.406+0.000 sys 19.375+0.000)
More information about the Mercurial-devel
mailing list