D4371: treemanifest: use visitchildrenset when doing a walk
spectral (Kyle Lippincott)
phabricator at mercurial-scm.org
Fri Aug 24 05:36:44 UTC 2018
spectral created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.
REVISION SUMMARY
For this series, starting at 'introduce lazy loading of subdirs' and ending with
this commit, we get the following timing numbers, using roughly the same
methodology and setup that we did in an earlier commit.
"before" is https://phab.mercurial-scm.org/rHGc62184c6299c09d2e8e7be340f9aee138229cb86 + https://phab.mercurial-scm.org/D4364 + https://phab.mercurial-scm.org/D4365.
diff --git:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 1.604 s +- 0.035 s | 1.604 s +- 0.028 s | 100.0%
m-u | | x | 1.583 s +- 0.049 s | 1.610 s +- 0.030 s | 101.7%
m-u | x | | 1.599 s +- 0.023 s | 1.595 s +- 0.031 s | 99.7%
m-u | x | x | 107.4 ms +- 1.5 ms | 107.6 ms +- 2.0 ms | 100.2%
l-d-r | | | 236.6 ms +- 6.4 ms | 235.9 ms +- 8.0 ms | 99.7%
l-d-r | | x | 241.2 ms +- 5.0 ms | 240.4 ms +- 6.4 ms | 99.7%
l-d-r | x | | 117.8 ms +- 2.0 ms | 120.2 ms +- 4.8 ms | 102.0%
l-d-r | x | x | 118.4 ms +- 2.0 ms | 119.1 ms +- 2.1 ms | 100.6%
citc1 | x | x | 480.1 ms +- 8.3 ms | 482.7 ms +- 9.4 ms | 100.5%
citc2 | x | x | 490.5 ms +- 13.3 ms | 482.4 ms +- 7.5 ms | 98.3%
diff -c . --git:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 352.5 ms +- 7.6 ms | 342.7 ms +- 6.7 ms | 97.2%
m-u | | x | 207.4 ms +- 4.9 ms | 205.6 ms +- 4.0 ms | 99.1%
m-u | x | | 348.3 ms +- 7.4 ms | 343.4 ms +- 4.9 ms | 98.6%
m-u | x | x | 162.2 ms +- 1.4 ms | 163.2 ms +- 3.0 ms | 100.6%
l-d-r | | | 98.0 ms +- 2.4 ms | 98.2 ms +- 2.6 ms | 100.2%
l-d-r | | x | 5.117 s +- 0.050 s | 4.353 s +- 0.055 s | 85.1% <--
l-d-r | x | | 98.7 ms +- 3.2 ms | 99.7 ms +- 2.7 ms | 101.0%
l-d-r | x | x | 1.008 s +- 0.016 s | 878.7 ms +- 11.6 ms | 87.2% <--
citc1 | x | x | 4.809 s +- 0.096 s | 4.512 s +- 0.084 s | 93.8% <--
citc2 | x | x | 636.0 ms +- 9.4 ms | 635.6 ms +- 7.4 ms | 99.9%
rebase -r . --keep -d .^^:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 6.805 s +- 0.241 s | 6.666 s +- 0.089 s | 98.0%
m-u | | x | 6.693 s +- 0.159 s | 6.684 s +- 0.101 s | 99.9%
m-u | x | | 6.679 s +- 0.167 s | 6.668 s +- 0.341 s | 99.8%
m-u | x | x | 682.2 ms +- 55.5 ms | 660.0 ms +- 56.7 ms | 96.7%
l-d-r | | | 769.5 ms +- 18.1 ms | 770.4 ms +- 11.3 ms | 100.1%
l-d-r | | x | 7.439 s +- 0.077 s | 6.983 s +- 0.047 s | 93.9% <--
l-d-r | x | | 332.4 ms +- 19.0 ms | 324.9 ms +- 3.8 ms | 97.7%
l-d-r | x | x | 5.894 s +- 0.139 s | 5.831 s +- 0.106 s | 98.9%
citc1 | x | x | 15.548 s +- 0.217 s | 13.604 s +- 0.745 s | 87.5% <--
citc2 | x | x | 1.185 s +- 0.383 s | 1.091 s +- 0.141 s | 92.1% <--
status --change . --copies:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 320.6 ms +- 8.2 ms | 318.9 ms +- 7.1 ms | 99.5%
m-u | | x | 182.1 ms +- 4.4 ms | 178.5 ms +- 2.8 ms | 98.0%
m-u | x | | 327.4 ms +- 8.3 ms | 323.1 ms +- 3.0 ms | 98.7%
m-u | x | x | 142.5 ms +- 2.6 ms | 142.2 ms +- 3.2 ms | 99.8%
l-d-r | | | 95.5 ms +- 1.6 ms | 94.6 ms +- 1.7 ms | 99.1%
l-d-r | | x | 5.065 s +- 0.076 s | 4.141 s +- 0.038 s | 81.8% <--
l-d-r | x | | 98.8 ms +- 1.8 ms | 98.5 ms +- 2.9 ms | 99.7%
l-d-r | x | x | 1.017 s +- 0.018 s | 875.4 ms +- 9.4 ms | 86.1% <--
citc1 | x | x | 4.575 s +- 0.082 s | 4.567 s +- 0.064 s | 99.8%
citc2 | x | x | 577.5 ms +- 15.7 ms | 582.4 ms +- 11.4 ms | 100.8%
status --copies:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 2.338 s +- 0.046 s | 2.348 s +- 0.042 s | 100.4%
m-u | | x | 2.375 s +- 0.036 s | 2.362 s +- 0.026 s | 99.5%
m-u | x | | 2.349 s +- 0.034 s | 2.352 s +- 0.029 s | 100.1%
m-u | x | x | 116.6 ms +- 1.5 ms | 115.9 ms +- 2.2 ms | 99.4%
l-d-r | | | 720.5 ms +- 10.5 ms | 716.2 ms +- 7.0 ms | 99.4%
l-d-r | | x | 729.0 ms +- 12.2 ms | 728.7 ms +- 9.3 ms | 100.0%
l-d-r | x | | 210.4 ms +- 2.2 ms | 212.2 ms +- 4.9 ms | 100.9%
l-d-r | x | x | 210.9 ms +- 5.3 ms | 209.1 ms +- 4.4 ms | 99.1%
citc1 | x | x | 612.9 ms +- 17.7 ms | 625.7 ms +- 15.7 ms | 102.1%
citc2 | x | x | 497.8 ms +- 12.7 ms | 498.2 ms +- 13.6 ms | 100.1%
update $rev^; ~/src/hg/hg{hg}/hg update $rev:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 3.912 s +- 0.044 s | 3.930 s +- 0.051 s | 100.5%
m-u | | x | 3.680 s +- 0.088 s | 3.649 s +- 0.057 s | 99.2%
m-u | x | | 3.898 s +- 0.042 s | 3.915 s +- 0.040 s | 100.4%
m-u | x | x | 397.5 ms +- 6.5 ms | 407.3 ms +- 17.0 ms | 102.5%
l-d-r | | | 538.6 ms +- 8.8 ms | 535.8 ms +- 8.1 ms | 99.5%
l-d-r | | x | 9.010 s +- 0.138 s | 9.006 s +- 0.030 s | 100.0%
l-d-r | x | | 307.6 ms +- 6.3 ms | 306.3 ms +- 3.0 ms | 99.6%
l-d-r | x | x | 1.792 s +- 0.010 s | 2.076 s +- 0.045 s | 115.8% <--
citc1 | x | x | 6.705 s +- 0.108 s | 6.532 s +- 0.096 s | 97.4%
citc2 | x | x | 1.213 s +- 0.049 s | 1.225 s +- 0.040 s | 101.0%
Unfortunately, like I mentioned in the commit at the start of this series, it
pains me to say that these numbers are *not reliable*. The 115.8% slowdown is
reproducibble when running under hyperfine, but I have been unable to reproduce them
myself. Every run I do with bobth before and after is ~2.0-2.2s on both
versions, this includes things like --color=never, >/dev/null, and using a
wrapper script that has reproduced outlier results like this in the past. I
don't know what it is about being run under hyperfine, but *something* is either
breaking or improving the behavior of the GC, I believe.
I do not believe that my code is actually the cause of the worsening of the
timing here, and am thus only relatively confident that it's providing the
highlighted perfomance *wins* either :)
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D4371
AFFECTED FILES
mercurial/manifest.py
CHANGE DETAILS
diff --git a/mercurial/manifest.py b/mercurial/manifest.py
--- a/mercurial/manifest.py
+++ b/mercurial/manifest.py
@@ -1033,19 +1033,22 @@
def _walk(self, match):
'''Recursively generates matching file names for walk().'''
- if not match.visitdir(self._dir[:-1] or '.'):
+ visit = match.visitchildrenset(self._dir[:-1] or '.')
+ if not visit:
return
# yield this dir's files and walk its submanifests
self._load()
+ visit = self._loadchildrensetlazy(visit)
for p in sorted(list(self._dirs) + list(self._files)):
if p in self._files:
fullp = self._subpath(p)
if match(fullp):
yield fullp
else:
- for f in self._dirs[p]._walk(match):
- yield f
+ if not visit or p[:-1] in visit:
+ for f in self._dirs[p]._walk(match):
+ yield f
def matches(self, match):
'''generate a new manifest filtered by the match argument'''
To: spectral, #hg-reviewers
Cc: mercurial-devel
More information about the Mercurial-devel
mailing list