[PATCH 0 of 3] 1286 regression fix and two extra small performance patches

[Christian Boos]

Petr Kodl wrote:
> Series of three patches improving mercurial performance on case folding systems.
>
> The first patch is a severe performance regression reported as 1286
> The second and third patch are both perfomance tweaks.
> _______________________________________________
>   

Hello,

I did some more performance tests for hg stat and diff earlier this day 
based on a previous version of your patches. The principal one, the 
third in this series ("clustered stat" patch) seems to be unchanged, so 
I'll present the results here.

The first experiment is to run `hg stat` and `hg diff` on a clean 
working copy of a 23,000 files repository. The working copy contains no 
changes to the versioned files and there are no extra, unversioned files 
in it.

 -------------------------------------------------------
 Revision                              || stat  || diff
 -------------------------------------------------------
 crew before 2c1f18b88b6a              || 4s    || 4s
 crew at 2c1f18b88b6a                  || 1s    || 2.8s
 clustered stat calls patch (96aa7a92) || 1s    || 1.1s
 issue1286 patch (7e5c9354)            || 0.77s || 0.88s
 eliminate normpath patch (0edb6b11)   || 0.58s || 0.66s
 -------------------------------------------------------
      Table A. Performance improvements on Windows


 -------------------------------------------------------
 Revision                              || stat  || diff
 -------------------------------------------------------
 crew at 2c1f18b88b6a                  || 0.65s || 0.85s
 clustered stat calls patch (96aa7a92) || 0.65s || 0.82s
 -------------------------------------------------------
      Table B. Performance status quo on Linux

The speed improvement is quite visible on Windows, as noted in earlier 
mails, and the last two patches improve things even more. On Linux, the 
cluster patch only makes a pretty small (but constant) difference, and 
the two last patches made no difference on that platform.


I also wanted to check my hypothesis that performance would eventually 
degrade quickly when there are extra, unversioned files in the working 
directory.
So I ran another experiment: for each file xyz in the manifest, I 
successively created extra files xyz.1, xyz.2, etc. and did some testing 
after each iterations with and without the "clustered stat" patch. The 
stat command used was `hg stat -q`, in order to ignore the unknown files.

 --------------------------------------------------
              || all but cluster||  cluster patch
 Extra Files  || stat  || diff  || stat  || diff
 ---------------------------------------------------
 0            || 2.86s || 2.94s || 0.57s || 0.64s (*)
 1            || 2.66s || 2.72s || 0.71s || 0.78s 
 2            || 2.67s || 2.74s || 0.90s || 0.96s
 3            || 2.68s || 2.77s || 1.03s || 1.11s
 ...
 6            || 2.92s || 3.00s || 1.48s || 1.56s
 ...
 8            || 2.92s || 3.00s || 1.86s || 1.95s
 ...
 10           || 2.91s || 2.98s || 2.93s || 3.01s
 --------------------------------------------------
  Table C. Impact of unversioned files, on Windows


 --------------------------------------------------
              || all but cluster||  cluster patch
 Extra Files  || stat  || diff  || stat  || diff
 --------------------------------------------------
 0            || 0.73s || 0.85s || 0.72s || 0.82s (*)
 1            || 0.73s || 0.85s || 1.02s || 1.14s
 2            || 0.73s || 0.85s || 1.33s || 1.46s
 3            || 0.74s || 0.85s || 1.65s || 1.77s
 4            || 0.74s || 0.85s || 1.95s || 2.07s
 --------------------------------------------------
  Table D. Impact of unversioned files, on Linux

With the results of Table D. I think it's clear that the "clustered 
stat" patch doesn't pay off on Linux. To the opposite, on Windows it 
seems extremely useful, and the performance degradation is not critical 
until there are at least 10 unversioned files per versioned file in the 
working directory.

But the "cluster patch" patch is nevertheless not OK as it is yet, even 
on Windows, as it seems that the cost of a whole working directory scan 
is implied even when an explicit path is given to  stat or diff.
I run `hg stat -q <path>` and `hg diff <path>` in a even more crowded 
working tree, containing 23k versioned files and 515k unversioned files, 
232k in src/ alone.

              || all but cluster||  cluster patch
 <path>       || stat  || diff  || stat  || diff
 --------------------------------------------------
 src          ||2.06s**|| 1.16s ||2.07s**|| 5.04s 
 src/dir/file || 0.29s || 0.36s || 4.86s || 4.96s 
 --------------------------------------------------
  Table E. cluster patch problem when given a path

This however sounds like a problem that could be easily fixed.


Some additional notes about unrelated findings:

(*) There's a strange difference between Table A. and C.'s first row 
(stat of 1s vs 2.86s), as well as between Table B. and D.'s first row 
(stat of 0.65s vs. 0.73s) as they should be the same. The only 
difference is that I have added the `-q` flag in the second experiment 
(using `-mard` gave the same timings as using `-q`). Most probably `hg 
stat` reverts in this case to the same strategy as the one used for `hg 
diff`, as the numbers in the stat and diff columns evolve in a similar 
way, in tables C. and D.

(**) In table E. `hg stat -u src` didn't really work, as no output was 
produced, but `hg st -nu | grep ^src | wc -l` says 232516. This works on 
Linux. Windows specific bug?


If nothing else, I hope my rough measurements will raise the attention 
on these patches, which could eventually manage to make hg run faster on 
Windows than it is on Linux, for big repositories :-)

-- Christian