[PATCH 0 of 3] Fast implemenation of parseindex in C

[Bernhard Leiner]

2008/10/7 Bernhard Leiner <mailinglists.bleiner at gmail.com>:
> 2008/10/6 Matt Mackall <mpm at selenic.com>:
>>
>> On Sun, 2008-09-21 at 22:24 +0200, Bernhard Leiner wrote:
>>> Ok, here is the perf extension with a new perfparseindex test.
>>>
>>> parseindex implementation in C:
>>> ! wall 0.032150 comb 0.030000 user 0.020000 sys 0.010000 (best of 100)
>>> vs. hg-stable:
>>> ! wall 0.065412 comb 0.060000 user 0.050000 sys 0.010000 (best of 100)
>>
>> (sorry, digging through mail backlog)
>>
>> Hmm, that's not as good as I hoped, but a good start! I was hoping to
>> see something like 5x when we got around to doing this. Perhaps there
>> are some more optimization opportunities.. In particular, Py_BuildObject
>> is quite helpful.

I can't see much room for optimizations in the C code any more, but I
noticed something odd... see below

>> It'd also be interesting to know what happens for very large indexes.

As example for large indexes I used a linux2.6 repo (00changelog.i and
00manifest.i size is about 7 MB, lazy parsing disabled). The results
from the perfparseindex tests are quite disappointing:

parseindex in C:
! wall 0.950195 comb 0.920000 user 0.870000 sys 0.050000 (best of 9)
vs. hg-stable:
! wall 1.541232 comb 1.510000 user 1.420000 sys 0.090000 (best of 6)

Since I didn't understand why the speedup for the large files is much
lower I wrote another performance testing script that really looks
_only_ on the raw performance of the index parsing (no hg overhead, no
disc I/O overhead, ...). Here are some results with different index
files. From a quite small one to a very large one:

size      C impl. speedup
    460      4.2
   1669      5.5
  64922      6.2
 109943      6.1
 418816      3.4  (no inlined data, this size was used in the perf
results sent earlier)
7092480      1.9  (no inlined data)

It looks like the speedup for small to medium size index files is
about 5 - 6 (like Matt expected/hoped) but the speedup gets lower when
the file gets bigger.

And so far I have no explanation why.

-- 
Bernhard Leiner           http://bernh.net