D4767: exchangev2: recognize narrow patterns when pulling
indygreg (Gregory Szorc)
phabricator at mercurial-scm.org
Wed Sep 26 21:45:21 UTC 2018
indygreg created this revision.
Herald added subscribers: mercurial-devel, mjpieters.
Herald added a reviewer: hg-reviewers.
REVISION SUMMARY
pulloperation instances were recently taught to record file
include and exclude patterns to facilitate narrow file transfer.
Teaching the exchangev2 code to transfer a subset of files is
as simple as constructing a narrow matcher from these patterns and
filtering all seen file paths through it.
In addition to testing presence of revlogs, we also test for presence
of copy/rename metadata when the source revision is outside the
narrow spec. This verifes that our narrow filelog class is working
properly.
Keep in mind that this change only influences file data: we're
still fetching all changeset and manifest data. So, there's still
a ton of "partial clone" to implement in exchangev2.
On a personal note, I derive gratification that this feature requires
very few lines of new code to implement. And notably, none of those
lines influence the parameters sent to the server: the client just
requests less repository data than it did before.
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D4767
AFFECTED FILES
mercurial/exchangev2.py
tests/test-fastcheckout.t
CHANGE DETAILS
diff --git a/tests/test-fastcheckout.t b/tests/test-fastcheckout.t
--- a/tests/test-fastcheckout.t
+++ b/tests/test-fastcheckout.t
@@ -3,6 +3,8 @@
$ cat >> $HGRCPATH << EOF
> [extensions]
> fastcheckout =
+ > [diff]
+ > git = true
> EOF
$ hg init server
@@ -32,8 +34,13 @@
$ hg commit -A -m 'commit 3'
adding dir0/file3.txt
+ $ hg cp dir1/file1.txt dir0/file1-copy.txt
+ $ hg commit -m 'commit 4'
+
$ hg log -G -T '{rev}:{node} {desc}'
- @ 3:27c5539872f8d2cfd5b95cab1fcaaee9b9bf9459 commit 3
+ @ 4:1816adf6a180913ad5db2a795fd09e02bd01657b commit 4
+ |
+ o 3:27c5539872f8d2cfd5b95cab1fcaaee9b9bf9459 commit 3
|
o 2:15c769176aab806212712ee9df382ebc7b63ade1 commit 2
|
@@ -159,18 +166,74 @@
narrow-dest0/.hg/store/data/dir0/child1/py1.py.i
narrow-dest0/.hg/store/data/dir0/child1/py2.py.i
narrow-dest0/.hg/store/data/dir0/file0.txt.i
- narrow-dest0/.hg/store/data/dir1/file1.txt.i
- narrow-dest0/.hg/store/data/dir1/file2.txt.i
- narrow-dest0/.hg/store/data/dir1/py3.py.i
- narrow-dest0/.hg/store/data/foo.i
narrow-dest0/.hg/store/fncache
narrow-dest0/.hg/store/narrowspec
narrow-dest0/.hg/store/phaseroots
narrow-dest0/.hg/store/undo
narrow-dest0/.hg/store/undo.backupfiles
narrow-dest0/.hg/store/undo.phaseroots
#endif
+Incremental pull fetches new files in narrow spec
+
+ $ hg fastcheckout http://localhost:$HGPORT 27c5539872 narrow-dest0
+ fetching data from http://localhost:$HGPORT
+ searching for changes
+ new changesets 27c5539872f8
+ updating to 27c5539872f8
+ 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+
+#if reporevlogstore
+ $ find narrow-dest0/.hg/store -type f | sort
+ narrow-dest0/.hg/store/00changelog.i
+ narrow-dest0/.hg/store/00manifest.i
+ narrow-dest0/.hg/store/data/dir0/child0/c.c.i
+ narrow-dest0/.hg/store/data/dir0/child0/py0.py.i
+ narrow-dest0/.hg/store/data/dir0/child1/py1.py.i
+ narrow-dest0/.hg/store/data/dir0/child1/py2.py.i
+ narrow-dest0/.hg/store/data/dir0/file0.txt.i
+ narrow-dest0/.hg/store/data/dir0/file3.txt.i
+ narrow-dest0/.hg/store/fncache
+ narrow-dest0/.hg/store/narrowspec
+ narrow-dest0/.hg/store/phaseroots
+ narrow-dest0/.hg/store/undo
+ narrow-dest0/.hg/store/undo.backup.fncache
+ narrow-dest0/.hg/store/undo.backup.phaseroots
+ narrow-dest0/.hg/store/undo.backupfiles
+ narrow-dest0/.hg/store/undo.phaseroots
+
+#endif
+
+Rename metadata for file outside of narrow spec is silently dropped
+
+ $ hg fastcheckout http://localhost:$HGPORT 1816adf6a18 narrow-dest0
+ fetching data from http://localhost:$HGPORT
+ searching for changes
+ new changesets 1816adf6a180
+ updating to 1816adf6a180
+ 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+
+ $ hg --cwd narrow-dest0 diff -c .
+ diff --git a/dir0/file1-copy.txt b/dir0/file1-copy.txt
+ new file mode 100644
+ --- /dev/null
+ +++ b/dir0/file1-copy.txt
+ @@ -0,0 +1,1 @@
+ +0
+
+Rename metadata for file inside narrow spec is reported
+
+ $ hg fastcheckout http://localhost:$HGPORT 1816adf6a18 rename-included
+ fetching data from http://localhost:$HGPORT
+ new changesets 6a9937924083:1816adf6a180
+ updating to 1816adf6a180
+ 11 files updated, 0 files merged, 0 files removed, 0 files unresolved
+
+ $ hg --cwd rename-included diff -c .
+ diff --git a/dir1/file1.txt b/dir0/file1-copy.txt
+ copy from dir1/file1.txt
+ copy to dir0/file1-copy.txt
+
Should not have any server-side errors
$ cat error.log
diff --git a/mercurial/exchangev2.py b/mercurial/exchangev2.py
--- a/mercurial/exchangev2.py
+++ b/mercurial/exchangev2.py
@@ -19,6 +19,7 @@
bookmarks,
error,
mdiff,
+ narrowspec,
phases,
pycompat,
setdiscovery,
@@ -30,6 +31,14 @@
remote = pullop.remote
tr = pullop.trmanager.transaction()
+ # We don't use the repo's narrow matcher here because the patterns passed
+ # to exchange.pull() could be different.
+ narrowmatcher = narrowspec.match(repo.root,
+ # Empty maps to nevermatcher. So always
+ # set includes if missing.
+ pullop.includepats or {'path:.'},
+ pullop.excludepats)
+
# Figure out what needs to be fetched.
common, fetch, remoteheads = _pullchangesetdiscovery(
repo, remote, pullop.heads, abortwhenunrelated=pullop.force)
@@ -63,7 +72,7 @@
# Find all file nodes referenced by added manifests and fetch those
# revisions.
- fnodes = _derivefilesfrommanifests(repo, manres['added'])
+ fnodes = _derivefilesfrommanifests(repo, narrowmatcher, manres['added'])
_fetchfiles(repo, tr, remote, fnodes, manres['linkrevs'])
def _pullchangesetdiscovery(repo, remote, heads, abortwhenunrelated=True):
@@ -311,7 +320,7 @@
'linkrevs': linkrevs,
}
-def _derivefilesfrommanifests(repo, manifestnodes):
+def _derivefilesfrommanifests(repo, matcher, manifestnodes):
"""Determine what file nodes are relevant given a set of manifest nodes.
Returns a dict mapping file paths to dicts of file node to first manifest
@@ -332,7 +341,8 @@
md = m.readfast()
for path, fnode in md.items():
- fnodes[path].setdefault(fnode, manifestnode)
+ if matcher(path):
+ fnodes[path].setdefault(fnode, manifestnode)
return fnodes
To: indygreg, #hg-reviewers
Cc: mjpieters, mercurial-devel
More information about the Mercurial-devel
mailing list