[PATCH 4 of 5 "] hgtagsfnodescache: inherit fnode from parent when possible
Pierre-Yves David
pierre-yves.david at ens-lyon.org
Mon Mar 11 05:24:03 EDT 2019
# HG changeset patch
# User Pierre-Yves David <pierre-yves.david at octobus.net>
# Date 1552263020 -3600
# Mon Mar 11 01:10:20 2019 +0100
# Node ID 988da9c806a11bc735adee84823b4eee7a8a0f9c
# Parent 85680409790f231f5865d76dab00efc0ccd42901
# EXP-Topic fnodecache
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 988da9c806a1
hgtagsfnodescache: inherit fnode from parent when possible
If a changeset does not update the content of `.hgtags`, it means it will use
the same file-node (for `.hgtags`) than its parents. In this such case we can
directly reuse the parent's file-node.
We use this property when updating the `hgtagsfnodescache` taking a faster path
if we already have a cached value for the parent's of the node we are looking
at.
Doing so provide a large performance boost when looking at a lot of fnode,
especially on repository with very large manifest:
timing for `tagsmod.fnoderevs(ui, repo, repo.changelog.revs())`
mercurial: (41907 revisions, 1923 files)
before: 6.9 seconds
after: 2.7 seconds (-54%)
pypy: (96266 revisions, 5198 files)
before: 80 seconds
after: 20 seconds (-75%)
mozilla-central: (463411 revisions, 272080 files)
before: 7166.4 seconds
after: 47.8 seconds (-99%, x150 speedup)
On a copy of mozilla-try with about 35K heads ans 1.7M changesets, this move
the computation from many hours to a couple of minutes. Making it more
interresting to do a full warm up of this cache before computing tags (from a
cold cache).
There seems to be other performance low hanging fruits, like avoid the used of
changectx or a more revision centric logic. However, the new code is fast enough
for my needs right now.
diff --git a/mercurial/tags.py b/mercurial/tags.py
--- a/mercurial/tags.py
+++ b/mercurial/tags.py
@@ -18,6 +18,7 @@ from .node import (
bin,
hex,
nullid,
+ nullrev,
short,
)
from .i18n import _
@@ -718,12 +719,33 @@ class hgtagsfnodescache(object):
if not computemissing:
return None
- # Populate missing entry.
- try:
- fnode = ctx.filenode('.hgtags')
- except error.LookupError:
- # No .hgtags file on this revision.
- fnode = nullid
+ fnode = None
+ cl = self._repo.changelog
+ p1rev, p2rev = cl._uncheckedparentrevs(rev)
+ p1node = cl.node(p1rev)
+ p1fnode = self.getfnode(p1node, computemissing=False)
+ if p2rev != nullrev:
+ # There is some no-merge changeset where p1 is null and p2 is set
+ # Processing them are merge is just slower, but still give a good
+ # result.
+ p2node = cl.node(p1rev)
+ p2fnode = self.getfnode(p2node, computemissing=False)
+ if p1fnode != p2fnode:
+ # we cannot rely on readfast because we don't know against what
+ # parent the readfast delta is computed
+ p1fnode = None
+ if p1fnode is not None:
+ mctx = ctx.manifestctx()
+ fnode = mctx.readfast().get('.hgtags')
+ if fnode is None:
+ fnode = p1fnode
+ if fnode is None:
+ # Populate missing entry.
+ try:
+ fnode = ctx.filenode('.hgtags')
+ except error.LookupError:
+ # No .hgtags file on this revision.
+ fnode = nullid
self._writeentry(offset, properprefix, fnode)
return fnode
diff --git a/tests/test-tags.t b/tests/test-tags.t
--- a/tests/test-tags.t
+++ b/tests/test-tags.t
@@ -145,7 +145,7 @@ Tag cache debug info written to blackbox
$ hg blackbox -l 6
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> identify
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> writing 48 bytes to cache/hgtagsfnodes1
- 1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> 0/1 cache hits/lookups in * seconds (glob)
+ 1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> 0/2 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> identify exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> blackbox -l 6
@@ -159,7 +159,7 @@ Failure to acquire lock results in no wr
$ hg blackbox -l 6
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> identify
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> not writing .hg/cache/hgtagsfnodes1 because lock cannot be acquired
- 1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> 0/1 cache hits/lookups in * seconds (glob)
+ 1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> 0/2 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> identify exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @b9154636be938d3d431e75a7c906504a079bfe07 (5000)> blackbox -l 6
@@ -363,7 +363,7 @@ Extra junk data at the end should get ov
$ hg blackbox -l 6
1970/01/01 00:00:00 bob @8dbfe60eff306a54259cfe007db9e330e7ecf866 (5000)> tags
1970/01/01 00:00:00 bob @8dbfe60eff306a54259cfe007db9e330e7ecf866 (5000)> writing 24 bytes to cache/hgtagsfnodes1
- 1970/01/01 00:00:00 bob @8dbfe60eff306a54259cfe007db9e330e7ecf866 (5000)> 2/3 cache hits/lookups in * seconds (glob)
+ 1970/01/01 00:00:00 bob @8dbfe60eff306a54259cfe007db9e330e7ecf866 (5000)> 3/4 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @8dbfe60eff306a54259cfe007db9e330e7ecf866 (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @8dbfe60eff306a54259cfe007db9e330e7ecf866 (5000)> tags exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @8dbfe60eff306a54259cfe007db9e330e7ecf866 (5000)> blackbox -l 6
@@ -384,7 +384,7 @@ Errors writing to .hgtags fnodes cache a
$ hg blackbox -l 6
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> couldn't write cache/hgtagsfnodes1: [Errno *] * (glob)
- 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 2/3 cache hits/lookups in * seconds (glob)
+ 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 3/4 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> blackbox -l 6
@@ -399,7 +399,7 @@ Errors writing to .hgtags fnodes cache a
$ hg blackbox -l 6
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing 24 bytes to cache/hgtagsfnodes1
- 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 2/3 cache hits/lookups in * seconds (glob)
+ 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 3/4 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> blackbox -l 6
@@ -427,7 +427,7 @@ Stripping doesn't truncate the tags cach
$ hg blackbox -l 5
1970/01/01 00:00:00 bob @0c192d7d5e6b78a714de54a2e9627952a877e25a (5000)> writing 24 bytes to cache/hgtagsfnodes1
- 1970/01/01 00:00:00 bob @0c192d7d5e6b78a714de54a2e9627952a877e25a (5000)> 2/3 cache hits/lookups in * seconds (glob)
+ 1970/01/01 00:00:00 bob @0c192d7d5e6b78a714de54a2e9627952a877e25a (5000)> 2/4 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @0c192d7d5e6b78a714de54a2e9627952a877e25a (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @0c192d7d5e6b78a714de54a2e9627952a877e25a (5000)> tags exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @0c192d7d5e6b78a714de54a2e9627952a877e25a (5000)> blackbox -l 5
@@ -445,7 +445,7 @@ Stripping doesn't truncate the tags cach
$ hg blackbox -l 6
1970/01/01 00:00:00 bob @035f65efb448350f4772141702a81ab1df48c465 (5000)> tags
1970/01/01 00:00:00 bob @035f65efb448350f4772141702a81ab1df48c465 (5000)> writing 24 bytes to cache/hgtagsfnodes1
- 1970/01/01 00:00:00 bob @035f65efb448350f4772141702a81ab1df48c465 (5000)> 2/3 cache hits/lookups in * seconds (glob)
+ 1970/01/01 00:00:00 bob @035f65efb448350f4772141702a81ab1df48c465 (5000)> 3/4 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @035f65efb448350f4772141702a81ab1df48c465 (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @035f65efb448350f4772141702a81ab1df48c465 (5000)> tags exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @035f65efb448350f4772141702a81ab1df48c465 (5000)> blackbox -l 6
More information about the Mercurial-devel
mailing list