D5169: exchangev2: support fetching shallow files history
indygreg (Gregory Szorc)
phabricator at mercurial-scm.org
Fri Oct 19 17:02:31 UTC 2018
indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.
REVISION SUMMARY
This commit teaches the exchangev2 client code to handle fetching shallow
files data.
Only shallow fetching of files data is supported: shallow fetching of
changeset and manifest data is explicitly not yet supported.
Previously, we would fetch file revisions for changesets that were received
by the current pull operation. In the new model, we calculate the set of
"relevant" changesets given the pull depth and only fetch files data for
those changesets.
We also teach the "filesdata" command invocation to vary parameters as needed.
The implementation here is far from complete or optimal. Subsequent pulls will
end up re-fetching a lot of files data. But the application of this data should
mostly be a no-op on the client, so it isn't a big deal.
Depending on the order file revisions are fetched in, revisions could get
inserted with the wrong revision number relationships. I think the best way
to deal with this is to remove revision numbers from storage and to either
dynamically derive them (by reconstructing a DAG from nodes/parents) or remove
revision numbers from the file storage interface completely.
A missing API that we'll likely want to write pretty soon is "ensure files
for revision(s) are present." We can kind of cajole exchangev2.pull() to do
this. But it isn't very efficient. For example, in simple cases like
widening the store to obtain data for a single revision, it is probably
more efficient to walk the manifest and find exactly which file revisions
are missing and to make explicit requests for just their data. In more
advanced cases, asking the server for all files data may be more efficient,
even though it requires sending data the client already has. There is tons
of room for future experimentation here. And TBH I'm not sure what the
final state will be.
Anyway, this commit gets us pretty close to being able to have shallow
and narrow checkouts with exchangev2/sqlite storage. Close enough that a
minimal extension should be able to provide fill in the gaps until the code
in core stabilizes and there is a user-facing way to trigger the
narrow/shallow bits from `hg clone` without also implying using of the
narrow extension...
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D5169
AFFECTED FILES
mercurial/exchangev2.py
tests/pullext.py
tests/test-wireproto-exchangev2-shallow.t
tests/test-wireproto-exchangev2.t
CHANGE DETAILS
diff --git a/tests/test-wireproto-exchangev2.t b/tests/test-wireproto-exchangev2.t
--- a/tests/test-wireproto-exchangev2.t
+++ b/tests/test-wireproto-exchangev2.t
@@ -1145,3 +1145,94 @@
client-stream-2/.hg/store/data/dir0/c.i
client-stream-2/.hg/store/data/dir0/d.i
#endif
+
+Shallow clone doesn't work with revlogs
+
+ $ hg --debug --config extensions.pullext=$TESTDIR/pullext.py clone --depth 1 -U http://localhost:$HGPORT client-shallow-revlogs
+ using http://localhost:$HGPORT/
+ sending capabilities command
+ query 1; heads
+ sending 2 commands
+ sending command heads: {}
+ sending command known: {
+ 'nodes': []
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=22; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ received frame(size=11; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=1; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=3; stream=2; streamflags=; type=command-response; flags=eos)
+ sending 1 commands
+ sending command changesetdata: {
+ 'fields': set([
+ 'bookmarks',
+ 'parents',
+ 'phase',
+ 'revision'
+ ]),
+ 'revisions': [
+ {
+ 'heads': [
+ '\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.'
+ ],
+ 'roots': [],
+ 'type': 'changesetdagrange'
+ }
+ ]
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=783; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ add changeset 3390ef850073
+ add changeset b709380892b1
+ add changeset 47fe012ab237
+ add changeset 97765fc3cd62
+ checking for updated bookmarks
+ sending 1 commands
+ sending command manifestdata: {
+ 'fields': set([
+ 'parents',
+ 'revision'
+ ]),
+ 'haveparents': True,
+ 'nodes': [
+ '\x99/Gy\x02\x9a=\xf8\xd0fm\x00\xbb\x92OicN&A',
+ '|2 \x1a\xa3\xa1R\xa9\xe6\xa9"+?\xa8\xd0\xe3\x0f\xc2V\xe8',
+ '\x8d\xd0W<\x7f\xaf\xe2\x04F\xcc\xea\xac\x05N\xea\xa4x\x91M\xdb',
+ '113\x85\xf2!\x8b\x08^\xb2Z\x821\x1e*\xdd\x0e\xeb\x8c3'
+ ],
+ 'tree': ''
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=967; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ sending 1 commands
+ sending command filesdata: {
+ 'fields': set([
+ 'linknode',
+ 'parents',
+ 'revision'
+ ]),
+ 'haveparents': False,
+ 'revisions': [
+ {
+ 'nodes': [
+ '\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.'
+ ],
+ 'type': 'changesetexplicit'
+ }
+ ]
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=1005; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ transaction abort!
+ rollback completed
+ (sent 5 HTTP requests and * bytes; received * bytes in responses) (glob)
+ abort: revlog storage does not support missing parents write mode
+ [255]
diff --git a/tests/test-wireproto-exchangev2-shallow.t b/tests/test-wireproto-exchangev2-shallow.t
new file mode 100644
--- /dev/null
+++ b/tests/test-wireproto-exchangev2-shallow.t
@@ -0,0 +1,568 @@
+#require sqlite
+
+Tests for wire protocol version 2 exchange.
+Tests in this file should be folded into existing tests once protocol
+v2 has enough features that it can be enabled via #testcase in existing
+tests.
+
+ $ . $TESTDIR/wireprotohelpers.sh
+ $ enablehttpv2client
+ $ cat >> $HGRCPATH << EOF
+ > [extensions]
+ > sqlitestore =
+ > pullext = $TESTDIR/pullext.py
+ > [storage]
+ > new-repo-backend=sqlite
+ > EOF
+
+Configure a server
+
+ $ hg init server-basic
+ $ enablehttpv2 server-basic
+ $ cd server-basic
+ $ mkdir dir0 dir1
+ $ echo a0 > a
+ $ echo b0 > b
+ $ hg -q commit -A -m 'commit 0'
+ $ echo c0 > dir0/c
+ $ echo d0 > dir0/d
+ $ hg -q commit -A -m 'commit 1'
+ $ echo e0 > dir1/e
+ $ echo f0 > dir1/f
+ $ hg -q commit -A -m 'commit 2'
+ $ echo c1 > dir0/c
+ $ echo e1 > dir1/e
+ $ hg commit -m 'commit 3'
+ $ echo c2 > dir0/c
+ $ echo e2 > dir1/e
+ $ echo f1 > dir1/f
+ $ hg commit -m 'commit 4'
+ $ echo a1 > a
+ $ echo b1 > b
+ $ hg commit -m 'commit 5'
+
+ $ hg log -G -T '{node} {desc}'
+ @ 93a8bd067ed2840d9aa810ad598168383a3a2c3a commit 5
+ |
+ o dc666cf9ecf3d94e6b830f30e5f1272e2a9164d9 commit 4
+ |
+ o 97765fc3cd624fd1fa0176932c21ffd16adf432e commit 3
+ |
+ o 47fe012ab237a8c7fc0c78f9f26d5866eef3f825 commit 2
+ |
+ o b709380892b193c1091d3a817f706052e346821b commit 1
+ |
+ o 3390ef850073fbc2f0dfff2244342c8e9229013a commit 0
+
+ $ hg serve -p $HGPORT -d --pid-file hg.pid -E error.log
+ $ cat hg.pid > $DAEMON_PIDS
+
+ $ cd ..
+
+Shallow clone pulls down latest revision of every file
+
+ $ hg --debug clone --depth 1 http://localhost:$HGPORT client-shallow-1
+ using http://localhost:$HGPORT/
+ sending capabilities command
+ query 1; heads
+ sending 2 commands
+ sending command heads: {}
+ sending command known: {
+ 'nodes': []
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=22; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ received frame(size=11; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=1; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=3; stream=2; streamflags=; type=command-response; flags=eos)
+ sending 1 commands
+ sending command changesetdata: {
+ 'fields': set([
+ 'bookmarks',
+ 'parents',
+ 'phase',
+ 'revision'
+ ]),
+ 'revisions': [
+ {
+ 'heads': [
+ '\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:'
+ ],
+ 'roots': [],
+ 'type': 'changesetdagrange'
+ }
+ ]
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=1170; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ add changeset 3390ef850073
+ add changeset b709380892b1
+ add changeset 47fe012ab237
+ add changeset 97765fc3cd62
+ add changeset dc666cf9ecf3
+ add changeset 93a8bd067ed2
+ checking for updated bookmarks
+ sending 1 commands
+ sending command manifestdata: {
+ 'fields': set([
+ 'parents',
+ 'revision'
+ ]),
+ 'haveparents': True,
+ 'nodes': [
+ '\x99/Gy\x02\x9a=\xf8\xd0fm\x00\xbb\x92OicN&A',
+ '|2 \x1a\xa3\xa1R\xa9\xe6\xa9"+?\xa8\xd0\xe3\x0f\xc2V\xe8',
+ '\x8d\xd0W<\x7f\xaf\xe2\x04F\xcc\xea\xac\x05N\xea\xa4x\x91M\xdb',
+ '113\x85\xf2!\x8b\x08^\xb2Z\x821\x1e*\xdd\x0e\xeb\x8c3',
+ 'H]O\xc2`\xef\\\xb9\xc0p6\x88K\x00k\x11\x0ej\xdby',
+ '\xd9;\xc4\x0b\x0e*GMp\xee\xf7}^\x91/f\x7fSd\x83'
+ ],
+ 'tree': ''
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=1515; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ sending 1 commands
+ sending command filesdata: {
+ 'fields': set([
+ 'linknode',
+ 'parents',
+ 'revision'
+ ]),
+ 'haveparents': False,
+ 'revisions': [
+ {
+ 'nodes': [
+ '\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:'
+ ],
+ 'type': 'changesetexplicit'
+ }
+ ]
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=1005; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ updating the branch cache
+ new changesets 3390ef850073:93a8bd067ed2
+ updating to branch default
+ resolving manifests
+ branchmerge: False, force: False, partial: False
+ ancestor: 000000000000, local: 000000000000+, remote: 93a8bd067ed2
+ a: remote created -> g
+ getting a
+ b: remote created -> g
+ getting b
+ dir0/c: remote created -> g
+ getting dir0/c
+ dir0/d: remote created -> g
+ getting dir0/d
+ dir1/e: remote created -> g
+ getting dir1/e
+ dir1/f: remote created -> g
+ getting dir1/f
+ 6 files updated, 0 files merged, 0 files removed, 0 files unresolved
+ (sent 5 HTTP requests and * bytes; received * bytes in responses) (glob)
+
+ $ sqlite3 -line client-shallow-1/.hg/store/db.sqlite << EOF
+ > SELECT id, path, revnum, node, p1rev, p2rev, linkrev, flags FROM filedata ORDER BY id ASC;
+ > EOF
+ id = 1
+ path = a
+ revnum = 0
+ node = \x9a8\x12)\x97\xb3\xac\x97\xbe*\x9a\xa2\xe5V\x83\x83A\xfd\xf2\xcc (esc)
+ p1rev = -1
+ p2rev = -1
+ linkrev = 5
+ flags = 2
+
+ id = 2
+ path = b
+ revnum = 0
+ node = \xb1zk\xd3g=\x9a\xb8\xce\xd5\x81\xa2 \xf6/=\xa5\xccEx (esc)
+ p1rev = -1
+ p2rev = -1
+ linkrev = 5
+ flags = 2
+
+ id = 3
+ path = dir0/c
+ revnum = 0
+ node = I\x1d\xa1\xbb\x89\xeax\xc0\xc0\xa2s[\x16\xce}\x93\x1d\xc8\xe2\r (esc)
+ p1rev = -1
+ p2rev = -1
+ linkrev = 4
+ flags = 2
+
+ id = 4
+ path = dir0/d
+ revnum = 0
+ node = S\x82\x06\xdc\x97\x1eR\x15@\xd6\x84:\xbf\xe6\xd1`2\xf6\xd4& (esc)
+ p1rev = -1
+ p2rev = -1
+ linkrev = 1
+ flags = 0
+
+ id = 5
+ path = dir1/e
+ revnum = 0
+ node = ]\xf3\xac\xd8\xd0\xc7\xfaP\x98\xd0'\x9a\x044\xc3\x02\x9e+x\xe1 (esc)
+ p1rev = -1
+ p2rev = -1
+ linkrev = 4
+ flags = 2
+
+ id = 6
+ path = dir1/f
+ revnum = 0
+ node = (\xc7v\xae\x08\xd0\xd5^\xb4\x06H\xb4\x01\xb9\x0f\xf5DH4\x8e (esc)
+ p1rev = -1
+ p2rev = -1
+ linkrev = 4
+ flags = 2
+
+Test a shallow clone with only some files
+
+ $ hg --debug clone --depth 1 --include dir0/ http://localhost:$HGPORT client-shallow-narrow-1
+ using http://localhost:$HGPORT/
+ sending capabilities command
+ query 1; heads
+ sending 2 commands
+ sending command heads: {}
+ sending command known: {
+ 'nodes': []
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=22; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ received frame(size=11; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=1; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=3; stream=2; streamflags=; type=command-response; flags=eos)
+ sending 1 commands
+ sending command changesetdata: {
+ 'fields': set([
+ 'bookmarks',
+ 'parents',
+ 'phase',
+ 'revision'
+ ]),
+ 'revisions': [
+ {
+ 'heads': [
+ '\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:'
+ ],
+ 'roots': [],
+ 'type': 'changesetdagrange'
+ }
+ ]
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=1170; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ add changeset 3390ef850073
+ add changeset b709380892b1
+ add changeset 47fe012ab237
+ add changeset 97765fc3cd62
+ add changeset dc666cf9ecf3
+ add changeset 93a8bd067ed2
+ checking for updated bookmarks
+ sending 1 commands
+ sending command manifestdata: {
+ 'fields': set([
+ 'parents',
+ 'revision'
+ ]),
+ 'haveparents': True,
+ 'nodes': [
+ '\x99/Gy\x02\x9a=\xf8\xd0fm\x00\xbb\x92OicN&A',
+ '|2 \x1a\xa3\xa1R\xa9\xe6\xa9"+?\xa8\xd0\xe3\x0f\xc2V\xe8',
+ '\x8d\xd0W<\x7f\xaf\xe2\x04F\xcc\xea\xac\x05N\xea\xa4x\x91M\xdb',
+ '113\x85\xf2!\x8b\x08^\xb2Z\x821\x1e*\xdd\x0e\xeb\x8c3',
+ 'H]O\xc2`\xef\\\xb9\xc0p6\x88K\x00k\x11\x0ej\xdby',
+ '\xd9;\xc4\x0b\x0e*GMp\xee\xf7}^\x91/f\x7fSd\x83'
+ ],
+ 'tree': ''
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=1515; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ sending 1 commands
+ sending command filesdata: {
+ 'fields': set([
+ 'linknode',
+ 'parents',
+ 'revision'
+ ]),
+ 'haveparents': False,
+ 'pathfilter': {
+ 'include': [
+ 'path:dir0'
+ ]
+ },
+ 'revisions': [
+ {
+ 'nodes': [
+ '\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:'
+ ],
+ 'type': 'changesetexplicit'
+ }
+ ]
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=355; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ updating the branch cache
+ new changesets 3390ef850073:93a8bd067ed2
+ updating to branch default
+ resolving manifests
+ branchmerge: False, force: False, partial: False
+ ancestor: 000000000000, local: 000000000000+, remote: 93a8bd067ed2
+ dir0/c: remote created -> g
+ getting dir0/c
+ dir0/d: remote created -> g
+ getting dir0/d
+ 2 files updated, 0 files merged, 0 files removed, 0 files unresolved
+ (sent 5 HTTP requests and * bytes; received * bytes in responses) (glob)
+
+ $ sqlite3 -line client-shallow-narrow-1/.hg/store/db.sqlite << EOF
+ > SELECT id, path, revnum, node, p1rev, p2rev, linkrev, flags FROM filedata ORDER BY id ASC;
+ > EOF
+ id = 1
+ path = dir0/c
+ revnum = 0
+ node = I\x1d\xa1\xbb\x89\xeax\xc0\xc0\xa2s[\x16\xce}\x93\x1d\xc8\xe2\r (esc)
+ p1rev = -1
+ p2rev = -1
+ linkrev = 4
+ flags = 2
+
+ id = 2
+ path = dir0/d
+ revnum = 0
+ node = S\x82\x06\xdc\x97\x1eR\x15@\xd6\x84:\xbf\xe6\xd1`2\xf6\xd4& (esc)
+ p1rev = -1
+ p2rev = -1
+ linkrev = 1
+ flags = 0
+
+Cloning an old revision with depth=1 works
+
+ $ hg --debug clone --depth 1 -r 97765fc3cd624fd1fa0176932c21ffd16adf432e http://localhost:$HGPORT client-shallow-2
+ using http://localhost:$HGPORT/
+ sending capabilities command
+ sending 1 commands
+ sending command lookup: {
+ 'key': '97765fc3cd624fd1fa0176932c21ffd16adf432e'
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=21; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ query 1; heads
+ sending 2 commands
+ sending command heads: {}
+ sending command known: {
+ 'nodes': []
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=22; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ received frame(size=11; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=1; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=3; stream=2; streamflags=; type=command-response; flags=eos)
+ sending 1 commands
+ sending command changesetdata: {
+ 'fields': set([
+ 'bookmarks',
+ 'parents',
+ 'phase',
+ 'revision'
+ ]),
+ 'revisions': [
+ {
+ 'heads': [
+ '\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.'
+ ],
+ 'roots': [],
+ 'type': 'changesetdagrange'
+ }
+ ]
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=783; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ add changeset 3390ef850073
+ add changeset b709380892b1
+ add changeset 47fe012ab237
+ add changeset 97765fc3cd62
+ checking for updated bookmarks
+ sending 1 commands
+ sending command manifestdata: {
+ 'fields': set([
+ 'parents',
+ 'revision'
+ ]),
+ 'haveparents': True,
+ 'nodes': [
+ '\x99/Gy\x02\x9a=\xf8\xd0fm\x00\xbb\x92OicN&A',
+ '|2 \x1a\xa3\xa1R\xa9\xe6\xa9"+?\xa8\xd0\xe3\x0f\xc2V\xe8',
+ '\x8d\xd0W<\x7f\xaf\xe2\x04F\xcc\xea\xac\x05N\xea\xa4x\x91M\xdb',
+ '113\x85\xf2!\x8b\x08^\xb2Z\x821\x1e*\xdd\x0e\xeb\x8c3'
+ ],
+ 'tree': ''
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=967; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ sending 1 commands
+ sending command filesdata: {
+ 'fields': set([
+ 'linknode',
+ 'parents',
+ 'revision'
+ ]),
+ 'haveparents': False,
+ 'revisions': [
+ {
+ 'nodes': [
+ '\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.'
+ ],
+ 'type': 'changesetexplicit'
+ }
+ ]
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=1005; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ updating the branch cache
+ new changesets 3390ef850073:97765fc3cd62
+ updating to branch default
+ resolving manifests
+ branchmerge: False, force: False, partial: False
+ ancestor: 000000000000, local: 000000000000+, remote: 97765fc3cd62
+ a: remote created -> g
+ getting a
+ b: remote created -> g
+ getting b
+ dir0/c: remote created -> g
+ getting dir0/c
+ dir0/d: remote created -> g
+ getting dir0/d
+ dir1/e: remote created -> g
+ getting dir1/e
+ dir1/f: remote created -> g
+ getting dir1/f
+ 6 files updated, 0 files merged, 0 files removed, 0 files unresolved
+ (sent 6 HTTP requests and * bytes; received * bytes in responses) (glob)
+
+Incremental pull of shallow clone fetches new changesets
+
+ $ hg --cwd client-shallow-2 --debug pull http://localhost:$HGPORT
+ pulling from http://localhost:$HGPORT/
+ using http://localhost:$HGPORT/
+ sending capabilities command
+ query 1; heads
+ sending 2 commands
+ sending command heads: {}
+ sending command known: {
+ 'nodes': [
+ '\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.'
+ ]
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=22; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ received frame(size=11; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=2; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=3; stream=2; streamflags=; type=command-response; flags=eos)
+ searching for changes
+ all local heads known remotely
+ sending 1 commands
+ sending command changesetdata: {
+ 'fields': set([
+ 'bookmarks',
+ 'parents',
+ 'phase',
+ 'revision'
+ ]),
+ 'revisions': [
+ {
+ 'heads': [
+ '\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:'
+ ],
+ 'roots': [
+ '\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.'
+ ],
+ 'type': 'changesetdagrange'
+ }
+ ]
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=400; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ add changeset dc666cf9ecf3
+ add changeset 93a8bd067ed2
+ checking for updated bookmarks
+ sending 1 commands
+ sending command manifestdata: {
+ 'fields': set([
+ 'parents',
+ 'revision'
+ ]),
+ 'haveparents': True,
+ 'nodes': [
+ 'H]O\xc2`\xef\\\xb9\xc0p6\x88K\x00k\x11\x0ej\xdby',
+ '\xd9;\xc4\x0b\x0e*GMp\xee\xf7}^\x91/f\x7fSd\x83'
+ ],
+ 'tree': ''
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=561; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ sending 1 commands
+ sending command filesdata: {
+ 'fields': set([
+ 'linknode',
+ 'parents',
+ 'revision'
+ ]),
+ 'haveparents': False,
+ 'revisions': [
+ {
+ 'nodes': [
+ '\xdcfl\xf9\xec\xf3\xd9Nk\x83\x0f0\xe5\xf1\'.*\x91d\xd9',
+ '\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:'
+ ],
+ 'type': 'changesetexplicit'
+ }
+ ]
+ }
+ received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
+ received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=1373; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ updating the branch cache
+ new changesets dc666cf9ecf3:93a8bd067ed2
+ (run 'hg update' to get a working copy)
+ (sent 5 HTTP requests and * bytes; received * bytes in responses) (glob)
+
+ $ hg --cwd client-shallow-2 up tip
+ merging dir0/c
+ merging dir1/e
+ 3 files updated, 2 files merged, 0 files removed, 0 files unresolved
diff --git a/tests/pullext.py b/tests/pullext.py
--- a/tests/pullext.py
+++ b/tests/pullext.py
@@ -10,6 +10,7 @@
from mercurial.i18n import _
from mercurial import (
commands,
+ error,
extensions,
localrepo,
repository,
@@ -19,6 +20,12 @@
if kwargs.get(r'include') or kwargs.get(r'exclude'):
kwargs[r'narrow'] = True
+ if kwargs.get(r'depth'):
+ try:
+ kwargs[r'depth'] = int(kwargs[r'depth'])
+ except ValueError:
+ raise error.Abort(_('--depth must be an integer'))
+
return orig(ui, repo, *args, **kwargs)
def featuresetup(ui, features):
@@ -28,11 +35,16 @@
entry = extensions.wrapcommand(commands.table, 'clone', clonecommand)
hasinclude = any(x[1] == 'include' for x in entry[1])
+ hasdepth = any(x[1] == 'depth' for x in entry[1])
if not hasinclude:
entry[1].append(('', 'include', [],
_('pattern of file/directory to clone')))
entry[1].append(('', 'exclude', [],
_('pattern of file/directory to not clone')))
+ if not hasdepth:
+ entry[1].append(('', 'depth', '',
+ _('ancestry depth of changesets to fetch')))
+
localrepo.featuresetupfuncs.add(featuresetup)
diff --git a/mercurial/exchangev2.py b/mercurial/exchangev2.py
--- a/mercurial/exchangev2.py
+++ b/mercurial/exchangev2.py
@@ -22,6 +22,7 @@
narrowspec,
phases,
pycompat,
+ repository,
setdiscovery,
)
@@ -91,6 +92,21 @@
manres = _fetchmanifests(repo, tr, remote, csetres['manifestnodes'])
+ # We don't properly support shallow changeset and manifest yet. So we apply
+ # depth limiting locally.
+ if pullop.depth:
+ relevantcsetnodes = set()
+ clnode = repo.changelog.node
+
+ for rev in repo.revs(b'ancestors(%ln, %d)',
+ pullheads, pullop.depth - 1):
+ relevantcsetnodes.add(clnode(rev))
+
+ csetrelevantfilter = lambda n: n in relevantcsetnodes
+
+ else:
+ csetrelevantfilter = lambda n: True
+
# If obtaining the raw store files, we need to scan the full repo to
# derive all the changesets, manifests, and linkrevs.
if usingrawchangelogandmanifest:
@@ -100,22 +116,27 @@
for rev in repo:
ctx = repo[rev]
+ node = ctx.node()
+
+ if not csetrelevantfilter(node):
+ continue
+
mnode = ctx.manifestnode()
- csetsforfiles.append(ctx.node())
+ csetsforfiles.append(node)
mnodesforfiles.append(mnode)
manifestlinkrevs[mnode] = rev
else:
- csetsforfiles = csetres['added']
+ csetsforfiles = [n for n in csetres['added'] if csetrelevantfilter(n)]
mnodesforfiles = manres['added']
manifestlinkrevs = manres['linkrevs']
# Find all file nodes referenced by added manifests and fetch those
# revisions.
fnodes = _derivefilesfrommanifests(repo, narrowmatcher, mnodesforfiles)
_fetchfilesfromcsets(repo, tr, remote, pathfilter, fnodes, csetsforfiles,
- manifestlinkrevs)
+ manifestlinkrevs, shallow=bool(pullop.depth))
def _checkuserawstorefiledata(pullop):
"""Check whether we should use rawstorefiledata command to retrieve data."""
@@ -564,7 +585,7 @@
weakref.proxy(tr))
def _fetchfilesfromcsets(repo, tr, remote, pathfilter, fnodes, csets,
- manlinkrevs):
+ manlinkrevs, shallow=False):
"""Fetch file data from explicit changeset revisions."""
def iterrevisions(objs, remaining, progress):
@@ -588,11 +609,16 @@
else:
continue
+ if b'linknode' in filerevision:
+ linknode = filerevision[b'linknode']
+ else:
+ linknode = node
+
yield (
node,
filerevision[b'parents'][0],
filerevision[b'parents'][1],
- node,
+ linknode,
basenode,
delta,
# Flags not yet supported.
@@ -609,6 +635,21 @@
commandmeta = remote.apidescriptor[b'commands'][b'filesdata']
batchsize = commandmeta.get(b'recommendedbatchsize', 50000)
+ shallowfiles = repository.REPO_FEATURE_SHALLOW_FILE_STORAGE in repo.features
+ fields = {b'parents', b'revision'}
+ clrev = repo.changelog.rev
+
+ # There are no guarantees that we'll have ancestor revisions if
+ # a) this repo has shallow file storage b) shallow data fetching is enabled.
+ # Force remote to not delta against possibly unknown revisions when these
+ # conditions hold.
+ haveparents = not (shallowfiles or shallow)
+
+ # Similarly, we may not have calculated linkrevs for all incoming file
+ # revisions. Ask the remote to do work for us in this case.
+ if not haveparents:
+ fields.add(b'linknode')
+
for i in pycompat.xrange(0, len(csets), batchsize):
batch = [x for x in csets[i:i + batchsize]]
if not batch:
@@ -620,8 +661,8 @@
b'type': b'changesetexplicit',
b'nodes': batch,
}],
- b'fields': {b'parents', b'revision'},
- b'haveparents': True,
+ b'fields': fields,
+ b'haveparents': haveparents,
}
if pathfilter:
@@ -643,7 +684,14 @@
fnode: manlinkrevs[mnode]
for fnode, mnode in fnodes[path].iteritems()}
+ def getlinkrev(node):
+ if node in linkrevs:
+ return linkrevs[node]
+ else:
+ return clrev(node)
+
store.addgroup(iterrevisions(objs, header[b'totalitems'],
progress),
- linkrevs.__getitem__,
- weakref.proxy(tr))
+ getlinkrev,
+ weakref.proxy(tr),
+ maybemissingparents=shallow)
To: indygreg, #hg-reviewers
Cc: mercurial-devel
More information about the Mercurial-devel
mailing list