D3535: wireprotov2: define and implement "filerevisionsslice" command
indygreg (Gregory Szorc)
phabricator at mercurial-scm.org
Fri May 11 22:35:57 UTC 2018
indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.
REVISION SUMMARY
This command can be thought of as a batch version of "filerevision."
Clients specify a changeset and a pair of paths. For each path within
that range, file revision data is retrieved.
While the new wire protocol is very efficient at processing thousands
of command requests, batch commands are often intrinsically more
efficient. In this particular case, this batch command prevents the
client from having to transmit potentially hundreds of thousands or
even millions of file names and file nodes. Instead, it can encapsulate
that request in a fraction of the wire size.
On the server side, batch requests such as this command allow the
server to perform a multi-get fetch from storage. Compared to a
wire protocol command per file, this should be much easier to
optimize and be far more efficient on servers.
This command is theoretically compatible with narrow clones composed
of any set of files. We don't to employ a matcher to limit which files
are retrieved because clients can construct an arbitrary combination of
"filerevisionsslice" commands to request the relevant set. That being
said, we may want to implement a filtering mechanism on this command
someday. I'm inclined to add this as a follow-up.
On the Firefox repository, a "filerevisionsslice" request for the full
file range takes a server ~33s CPU time on my i7-6700K. That's without
any optimizations. Profiling indicates most of that time is spent in
revlog code opening revlogs and resolving fulltext data. I have
experimented with formalizing a more advanced file storage interface
that supports multi-get and hooking a Redis cache into that interface
and the results are *very* promising. But ~33s CPU time to serve
effectively a full revision checkout is a pretty good place to start!
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D3535
AFFECTED FILES
mercurial/help/internals/wireprotocol.txt
mercurial/wireprotov2server.py
tests/test-http-protocol.t
tests/test-wireproto-command-capabilities.t
tests/test-wireproto-command-filerevisionsslice.t
CHANGE DETAILS
diff --git a/tests/test-wireproto-command-filerevisionsslice.t b/tests/test-wireproto-command-filerevisionsslice.t
new file mode 100644
--- /dev/null
+++ b/tests/test-wireproto-command-filerevisionsslice.t
@@ -0,0 +1,211 @@
+ $ . $TESTDIR/wireprotohelpers.sh
+
+ $ hg init server
+ $ enablehttpv2 server
+ $ cd server
+ $ echo a0 > a
+ $ echo b0 > b
+ $ mkdir dir0
+ $ echo c0 > dir0/c
+ $ echo d0 > dir0/d
+ $ echo e0 > dir0/e
+ $ echo z0 > z
+ $ hg -q commit -A -m 'commit 0'
+
+ $ echo a1 > a
+ $ echo b1 > b
+ $ hg commit -m 'commit 1'
+
+ $ echo ca > dir0/ca
+ $ echo dd > dir0/dd
+ $ hg -q commit -A -m 'commit 2'
+
+ $ hg mv a a-moved
+ $ hg commit -m 'commit 3'
+
+ $ hg log -T '{rev}:{node} {desc}\n'
+ 3:29c3cfed01c8dca4e2ed052998a0a4783a364265 commit 3
+ 2:4a8fbd921812449fb3cbaa4f8ef90082bee356d1 commit 2
+ 1:a168c837ab2d7935d50000845178380528427f01 commit 1
+ 0:fcf140e9e1ac1191ac879211b8918a79dfbb9f0d commit 0
+
+ $ hg serve -p $HGPORT -d --pid-file hg.pid -E error.log
+ $ cat hg.pid > $DAEMON_PIDS
+
+Request for unknown node fails
+
+ $ sendhttpv2peer << EOF
+ > command filerevisionsslice
+ > node aaaaaaaaaaaaaaaaaaaa
+ > startpath a
+ > endpath z
+ > EOF
+ creating http peer for wire protocol version 2
+ sending filerevisionsslice command
+ s> POST /api/exp-http-v2-0001/ro/filerevisionsslice HTTP/1.1\r\n
+ s> Accept-Encoding: identity\r\n
+ s> accept: application/mercurial-exp-framing-0005\r\n
+ s> content-type: application/mercurial-exp-framing-0005\r\n
+ s> content-length: 87\r\n
+ s> host: $LOCALIP:$HGPORT\r\n (glob)
+ s> user-agent: Mercurial debugwireproto\r\n
+ s> \r\n
+ s> O\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa3DnodeTaaaaaaaaaaaaaaaaaaaaGendpathAzIstartpathAaDnameRfilerevisionsslice
+ s> makefile('rb', None)
+ s> HTTP/1.1 200 OK\r\n
+ s> Server: testing stub value\r\n
+ s> Date: $HTTP_DATE$\r\n
+ s> Content-Type: application/mercurial-exp-framing-0005\r\n
+ s> Transfer-Encoding: chunked\r\n
+ s> \r\n
+ s> 36\r\n
+ s> .\x00\x00\x01\x00\x02\x012
+ s> \xa2Eerror\xa1GmessagePunknown revisionFstatusEerror
+ s> \r\n
+ received frame(size=46; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=eos)
+ s> 0\r\n
+ s> \r\n
+ response: [{b'error': {b'message': b'unknown revision'}, b'status': b'error'}]
+
+Fetching a single file works
+
+ $ sendhttpv2peer << EOF
+ > command filerevisionsslice
+ > node \xfc\xf1\x40\xe9\xe1\xac\x11\x91\xac\x87\x92\x11\xb8\x91\x8a\x79\xdf\xbb\x9f\x0d
+ > startpath z
+ > endpath z
+ > EOF
+ creating http peer for wire protocol version 2
+ sending filerevisionsslice command
+ s> POST /api/exp-http-v2-0001/ro/filerevisionsslice HTTP/1.1\r\n
+ s> Accept-Encoding: identity\r\n
+ s> accept: application/mercurial-exp-framing-0005\r\n
+ s> content-type: application/mercurial-exp-framing-0005\r\n
+ s> content-length: 87\r\n
+ s> host: $LOCALIP:$HGPORT\r\n (glob)
+ s> user-agent: Mercurial debugwireproto\r\n
+ s> \r\n
+ s> O\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa3DnodeT\xfc\xf1@\xe9\xe1\xac\x11\x91\xac\x87\x92\x11\xb8\x91\x8ay\xdf\xbb\x9f\rGendpathAzIstartpathAzDnameRfilerevisionsslice
+ s> makefile('rb', None)
+ s> HTTP/1.1 200 OK\r\n
+ s> Server: testing stub value\r\n
+ s> Date: $HTTP_DATE$\r\n
+ s> Content-Type: application/mercurial-exp-framing-0005\r\n
+ s> Transfer-Encoding: chunked\r\n
+ s> \r\n
+ s> 13\r\n
+ s> \x0b\x00\x00\x01\x00\x02\x011
+ s> \xa1FstatusBok
+ s> \r\n
+ received frame(size=11; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=continuation)
+ s> 43\r\n
+ s> ;\x00\x00\x01\x00\x02\x000
+ s> \xa1Jtotalitems\x01\xa3DnodeT)\xc0;\xbeA\x0e\xf7\x8e\xb2\xe2\x05\x90\xc1R\x88\xde\xb6\x9f)CDpathAzDsize\x03_Cz0\n
+ s> \xff
+ s> \r\n
+ received frame(size=59; request=1; stream=2; streamflags=; type=command-response; flags=)
+ s> 8\r\n
+ s> \x00\x00\x00\x01\x00\x02\x002
+ s> \r\n
+ s> 0\r\n
+ s> \r\n
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ response: [{b'status': b'ok'}, {b'totalitems': 1}, {b'node': b')\xc0;\xbeA\x0e\xf7\x8e\xb2\xe2\x05\x90\xc1R\x88\xde\xb6\x9f)C', b'path': b'z', b'size': 3}, bytearray['z0\n']]
+
+Fetching a range of files works
+
+ $ sendhttpv2peer << EOF
+ > command filerevisionsslice
+ > node \xa1\x68\xc8\x37\xab\x2d\x79\x35\xd5\x00\x00\x84\x51\x78\x38\x05\x28\x42\x7f\x01
+ > startpath b
+ > endpath dir0/d
+ > EOF
+ creating http peer for wire protocol version 2
+ sending filerevisionsslice command
+ s> POST /api/exp-http-v2-0001/ro/filerevisionsslice HTTP/1.1\r\n
+ s> Accept-Encoding: identity\r\n
+ s> accept: application/mercurial-exp-framing-0005\r\n
+ s> content-type: application/mercurial-exp-framing-0005\r\n
+ s> content-length: 92\r\n
+ s> host: $LOCALIP:$HGPORT\r\n (glob)
+ s> user-agent: Mercurial debugwireproto\r\n
+ s> \r\n
+ s> T\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa3DnodeT\xa1h\xc87\xab-y5\xd5\x00\x00\x84Qx8\x05(B\x7f\x01GendpathFdir0/dIstartpathAbDnameRfilerevisionsslice
+ s> makefile('rb', None)
+ s> HTTP/1.1 200 OK\r\n
+ s> Server: testing stub value\r\n
+ s> Date: $HTTP_DATE$\r\n
+ s> Content-Type: application/mercurial-exp-framing-0005\r\n
+ s> Transfer-Encoding: chunked\r\n
+ s> \r\n
+ s> 13\r\n
+ s> \x0b\x00\x00\x01\x00\x02\x011
+ s> \xa1FstatusBok
+ s> \r\n
+ received frame(size=11; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=continuation)
+ s> a9\r\n
+ s> \xa1\x00\x00\x01\x00\x02\x000
+ s> \xa1Jtotalitems\x03\xa3DnodeT\xb1zk\xd3g=\x9a\xb8\xce\xd5\x81\xa2\t\xf6/=\xa5\xccExDpathAbDsize\x03_Cb1\n
+ s> \xff\xa3DnodeT\x91DE4j\x0c\xa0b\x9b\xd4|\xeb]\xfe\x07\xe4\xd4\xcf%\x01DpathFdir0/cDsize\x03_Cc0\n
+ s> \xff\xa3DnodeTS\x82\x06\xdc\x97\x1eR\x15@\xd6\x84:\xbf\xe6\xd1`2\xf6\xd4&DpathFdir0/dDsize\x03_Cd0\n
+ s> \xff
+ s> \r\n
+ received frame(size=161; request=1; stream=2; streamflags=; type=command-response; flags=)
+ s> 8\r\n
+ s> \x00\x00\x00\x01\x00\x02\x002
+ s> \r\n
+ s> 0\r\n
+ s> \r\n
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ response: [{b'status': b'ok'}, {b'totalitems': 3}, {b'node': b'\xb1zk\xd3g=\x9a\xb8\xce\xd5\x81\xa2\t\xf6/=\xa5\xccEx', b'path': b'b', b'size': 3}, bytearray['b1\n'], {b'node': b'\x91DE4j\x0c\xa0b\x9b\xd4|\xeb]\xfe\x07\xe4\xd4\xcf%\x01', b'path': b'dir0/c', b'size': 3}, bytearray['c0\n'], {b'node': b'S\x82\x06\xdc\x97\x1eR\x15@\xd6\x84:\xbf\xe6\xd1`2\xf6\xd4&', b'path': b'dir0/d', b'size': 3}, bytearray['d0\n']]
+
+Copy metadata is included when appropriate
+
+ $ sendhttpv2peer << EOF
+ > command filerevisionsslice
+ > node \x29\xc3\xcf\xed\x01\xc8\xdc\xa4\xe2\xed\x05\x29\x98\xa0\xa4\x78\x3a\x36\x42\x65
+ > startpath a-moved
+ > endpath a-moved
+ > EOF
+ creating http peer for wire protocol version 2
+ sending filerevisionsslice command
+ s> POST /api/exp-http-v2-0001/ro/filerevisionsslice HTTP/1.1\r\n
+ s> Accept-Encoding: identity\r\n
+ s> accept: application/mercurial-exp-framing-0005\r\n
+ s> content-type: application/mercurial-exp-framing-0005\r\n
+ s> content-length: 99\r\n
+ s> host: $LOCALIP:$HGPORT\r\n (glob)
+ s> user-agent: Mercurial debugwireproto\r\n
+ s> \r\n
+ s> [\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa3DnodeT)\xc3\xcf\xed\x01\xc8\xdc\xa4\xe2\xed\x05)\x98\xa0\xa4x:6BeGendpathGa-movedIstartpathGa-movedDnameRfilerevisionsslice
+ s> makefile('rb', None)
+ s> HTTP/1.1 200 OK\r\n
+ s> Server: testing stub value\r\n
+ s> Date: $HTTP_DATE$\r\n
+ s> Content-Type: application/mercurial-exp-framing-0005\r\n
+ s> Transfer-Encoding: chunked\r\n
+ s> \r\n
+ s> 13\r\n
+ s> \x0b\x00\x00\x01\x00\x02\x011
+ s> \xa1FstatusBok
+ s> \r\n
+ received frame(size=11; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=continuation)
+ s> a9\r\n
+ s> \xa1\x00\x00\x01\x00\x02\x000
+ s> \xa1Jtotalitems\x01\xa4DnodeT4\x7f\xcdS\x0b\xe0J\x85\xe8EoM\xc1\x0b"\x16G#/\x84DpathGa-movedGrenamed\x82AaT\x9a8\x12)\x97\xb3\xac\x97\xbe*\x9a\xa2\xe5V\x83\x83A\xfd\xf2\xccDsize\x18A_XA\x01\n
+ s> copy: a\n
+ s> copyrev: 9a38122997b3ac97be2a9aa2e556838341fdf2cc\n
+ s> \x01\n
+ s> a1\n
+ s> \xff
+ s> \r\n
+ received frame(size=161; request=1; stream=2; streamflags=; type=command-response; flags=)
+ s> 8\r\n
+ s> \x00\x00\x00\x01\x00\x02\x002
+ s> \r\n
+ s> 0\r\n
+ s> \r\n
+ received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+ response: [{b'status': b'ok'}, {b'totalitems': 1}, {b'node': b'4\x7f\xcdS\x0b\xe0J\x85\xe8EoM\xc1\x0b"\x16G#/\x84', b'path': b'a-moved', b'renamed': [b'a', b'\x9a8\x12)\x97\xb3\xac\x97\xbe*\x9a\xa2\xe5V\x83\x83A\xfd\xf2\xcc'], b'size': 65}, bytearray['\x01\ncopy: a\ncopyrev: 9a38122997b3ac97be2a9aa2e556838341fdf2cc\n\x01\na1\n']]
+
+ $ cat error.log
diff --git a/tests/test-wireproto-command-capabilities.t b/tests/test-wireproto-command-capabilities.t
--- a/tests/test-wireproto-command-capabilities.t
+++ b/tests/test-wireproto-command-capabilities.t
@@ -202,8 +202,8 @@
s> Content-Type: application/mercurial-cbor\r\n
s> Content-Length: *\r\n (glob)
s> \r\n
- s> \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xaaEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
- cbor> {b'apibase': b'api/', b'apis': {b'exp-http-v2-0001': {b'commands': {b'branchmap': {b'args': {}, b'permissions': [b'pull']}, b'capabilities': {b'args': {}, b'permissions': [b'pull']}, b'filehistory': {b'args': {b'filenodebases': [b'deadbeef'], b'filenodeheads': [b'deadbeef'], b'path': b'path/to/file'}, b'permissions': [b'pull']}, b'filerevision': {b'args': {b'filenode': b'0123456789abcdef...', b'path': b'a/foo'}, b'permissions': [b'pull']}, b'heads': {b'args': {b'publiconly': False}, b'permissions': [b'pull']}, b'known': {b'args': {b'nodes': [b'deadbeef']}, b'permissions': [b'pull']}, b'listkeys': {b'args': {b'namespace': b'ns'}, b'permissions': [b'pull']}, b'lookup': {b'args': {b'key': b'foo'}, b'permissions': [b'pull']}, b'pushkey': {b'args': {b'key': b'key', b'namespace': b'ns', b'new': b'new', b'old': b'old'}, b'permissions': [b'push']}, b'rawstorefile': {b'args': {b'what': b'changelog'}, b'permissions': [b'pull']}}, b'compression': [{b'name': b'zlib'}], b'framingmediatypes': [b'application/mercurial-exp-framing-0005'], b'rawrepoformats': [b'generaldelta', b'revlogv1']}}, b'v1capabilities': b'batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash'}
+ s> \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xabEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullRfilerevisionsslice\xa2Dargs\xa3DnodeS0123456789abcdef...GendpathEb/barIstartpathEa/fooKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
+ cbor> {b'apibase': b'api/', b'apis': {b'exp-http-v2-0001': {b'commands': {b'branchmap': {b'args': {}, b'permissions': [b'pull']}, b'capabilities': {b'args': {}, b'permissions': [b'pull']}, b'filehistory': {b'args': {b'filenodebases': [b'deadbeef'], b'filenodeheads': [b'deadbeef'], b'path': b'path/to/file'}, b'permissions': [b'pull']}, b'filerevision': {b'args': {b'filenode': b'0123456789abcdef...', b'path': b'a/foo'}, b'permissions': [b'pull']}, b'filerevisionsslice': {b'args': {b'endpath': b'b/bar', b'node': b'0123456789abcdef...', b'startpath': b'a/foo'}, b'permissions': [b'pull']}, b'heads': {b'args': {b'publiconly': False}, b'permissions': [b'pull']}, b'known': {b'args': {b'nodes': [b'deadbeef']}, b'permissions': [b'pull']}, b'listkeys': {b'args': {b'namespace': b'ns'}, b'permissions': [b'pull']}, b'lookup': {b'args': {b'key': b'foo'}, b'permissions': [b'pull']}, b'pushkey': {b'args': {b'key': b'key', b'namespace': b'ns', b'new': b'new', b'old': b'old'}, b'permissions': [b'push']}, b'rawstorefile': {b'args': {b'what': b'changelog'}, b'permissions': [b'pull']}}, b'compression': [{b'name': b'zlib'}], b'framingmediatypes': [b'application/mercurial-exp-framing-0005'], b'rawrepoformats': [b'generaldelta', b'revlogv1']}}, b'v1capabilities': b'batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash'}
capabilities command returns expected info
@@ -227,7 +227,7 @@
s> Content-Type: application/mercurial-cbor\r\n
s> Content-Length: *\r\n (glob)
s> \r\n
- s> \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xaaEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
+ s> \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xabEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullRfilerevisionsslice\xa2Dargs\xa3DnodeS0123456789abcdef...GendpathEb/barIstartpathEa/fooKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
sending capabilities command
s> POST /api/exp-http-v2-0001/ro/capabilities HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
@@ -245,13 +245,13 @@
s> Content-Type: application/mercurial-exp-framing-0005\r\n
s> Transfer-Encoding: chunked\r\n
s> \r\n
- s> 2c1\r\n
- s> \xb9\x02\x00\x01\x00\x02\x012
- s> \xa1FstatusBok\xa4Hcommands\xaaEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005
+ s> 324\r\n
+ s> \x1c\x03\x00\x01\x00\x02\x012
+ s> \xa1FstatusBok\xa4Hcommands\xabEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullRfilerevisionsslice\xa2Dargs\xa3DnodeS0123456789abcdef...GendpathEb/barIstartpathEa/fooKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005
s> \r\n
- received frame(size=697; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=eos)
+ received frame(size=796; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=eos)
s> 0\r\n
s> \r\n
- response: [{b'status': b'ok'}, {b'commands': {b'branchmap': {b'args': {}, b'permissions': [b'pull']}, b'capabilities': {b'args': {}, b'permissions': [b'pull']}, b'filehistory': {b'args': {b'filenodebases': [b'deadbeef'], b'filenodeheads': [b'deadbeef'], b'path': b'path/to/file'}, b'permissions': [b'pull']}, b'filerevision': {b'args': {b'filenode': b'0123456789abcdef...', b'path': b'a/foo'}, b'permissions': [b'pull']}, b'heads': {b'args': {b'publiconly': False}, b'permissions': [b'pull']}, b'known': {b'args': {b'nodes': [b'deadbeef']}, b'permissions': [b'pull']}, b'listkeys': {b'args': {b'namespace': b'ns'}, b'permissions': [b'pull']}, b'lookup': {b'args': {b'key': b'foo'}, b'permissions': [b'pull']}, b'pushkey': {b'args': {b'key': b'key', b'namespace': b'ns', b'new': b'new', b'old': b'old'}, b'permissions': [b'push']}, b'rawstorefile': {b'args': {b'what': b'changelog'}, b'permissions': [b'pull']}}, b'compression': [{b'name': b'zlib'}], b'framingmediatypes': [b'application/mercurial-exp-framing-0005'], b'rawrepoformats': [b'generaldelta', b'revlogv1']}]
+ response: [{b'status': b'ok'}, {b'commands': {b'branchmap': {b'args': {}, b'permissions': [b'pull']}, b'capabilities': {b'args': {}, b'permissions': [b'pull']}, b'filehistory': {b'args': {b'filenodebases': [b'deadbeef'], b'filenodeheads': [b'deadbeef'], b'path': b'path/to/file'}, b'permissions': [b'pull']}, b'filerevision': {b'args': {b'filenode': b'0123456789abcdef...', b'path': b'a/foo'}, b'permissions': [b'pull']}, b'filerevisionsslice': {b'args': {b'endpath': b'b/bar', b'node': b'0123456789abcdef...', b'startpath': b'a/foo'}, b'permissions': [b'pull']}, b'heads': {b'args': {b'publiconly': False}, b'permissions': [b'pull']}, b'known': {b'args': {b'nodes': [b'deadbeef']}, b'permissions': [b'pull']}, b'listkeys': {b'args': {b'namespace': b'ns'}, b'permissions': [b'pull']}, b'lookup': {b'args': {b'key': b'foo'}, b'permissions': [b'pull']}, b'pushkey': {b'args': {b'key': b'key', b'namespace': b'ns', b'new': b'new', b'old': b'old'}, b'permissions': [b'push']}, b'rawstorefile': {b'args': {b'what': b'changelog'}, b'permissions': [b'pull']}}, b'compression': [{b'name': b'zlib'}], b'framingmediatypes': [b'application/mercurial-exp-framing-0005'], b'rawrepoformats': [b'generaldelta', b'revlogv1']}]
$ cat error.log
diff --git a/tests/test-http-protocol.t b/tests/test-http-protocol.t
--- a/tests/test-http-protocol.t
+++ b/tests/test-http-protocol.t
@@ -305,7 +305,7 @@
s> Content-Type: application/mercurial-cbor\r\n
s> Content-Length: *\r\n (glob)
s> \r\n
- s> \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xaaEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
+ s> \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xabEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullRfilerevisionsslice\xa2Dargs\xa3DnodeS0123456789abcdef...GendpathEb/barIstartpathEa/fooKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
sending heads command
s> POST /api/exp-http-v2-0001/ro/heads HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
diff --git a/mercurial/wireprotov2server.py b/mercurial/wireprotov2server.py
--- a/mercurial/wireprotov2server.py
+++ b/mercurial/wireprotov2server.py
@@ -608,6 +608,92 @@
return wireprototypes.v2streamingresponse(senddata(),
compressible=True)
+ at wireprotocommand('filerevisionsslice',
+ args={
+ 'node': b'0123456789abcdef...',
+ 'startpath': b'a/foo',
+ 'endpath': b'b/bar',
+ },
+ permission='pull')
+def filerevisionsslice(repo, proto, node, startpath, endpath):
+ # TODO consider advertising a limit for maximum slice size to limit work
+ # per invocation.
+
+ try:
+ ctx = repo[node]
+ except error.RepoLookupError:
+ return wireprototypes.v2errorresponse(
+ _('unknown revision'))
+
+ if startpath > endpath:
+ startpath, endpath = endpath, startpath
+
+ m = ctx.manifest()
+
+ # Find relevant files.
+ # TODO linear scans of manifests are bad. Manifest needs to grow an API
+ # that allows iteration starting from a key.
+
+ relevantpaths = []
+
+ for path, fnode in m.iteritems():
+ if path < startpath:
+ continue
+
+ relevantpaths.append((path, fnode))
+
+ if path >= endpath:
+ break
+
+ def senddata():
+ chunks = cborutil.streamencodemap({
+ 'totalitems': len(relevantpaths),
+ })
+ for chunk in chunks:
+ yield chunk
+
+ for path, fnode in relevantpaths:
+ res = {
+ 'path': path,
+ 'node': fnode,
+ }
+
+ fl = repo.file(path)
+ rev = fl.rev(fnode)
+
+ if fl.iscensored(rev):
+ res['censored'] = True
+ res['size'] = 0
+ for chunk in cborutil.streamencodemap(res):
+ yield chunk
+
+ # Always yield a bytestring for consistency.
+ for chunk in cborutil.streamencodebytestring(b''):
+ yield chunk
+
+ continue
+
+ # TODO support not having to buffer entire fulltext. But the
+ # file storage APIs don't support this yet, so...
+ fulltext = fl.revision(fnode)
+ res['size'] = len(fulltext)
+
+ renamed = fl.renamed(fnode)
+ if renamed:
+ res['renamed'] = renamed
+
+ for chunk in cborutil.streamencodemap(res):
+ yield chunk
+
+ # While our implementation must buffer in the fulltext, we shouldn't
+ # force this on the client. So split into an indefinite length
+ # bytestring and send those chunks.
+ for chunk in cborutil.streamencodeindefinitebytestring(fulltext):
+ yield chunk
+
+ return wireprototypes.v2streamingresponse(senddata(),
+ compressible=True)
+
@wireprotocommand('heads',
args={
'publiconly': False,
diff --git a/mercurial/help/internals/wireprotocol.txt b/mercurial/help/internals/wireprotocol.txt
--- a/mercurial/help/internals/wireprotocol.txt
+++ b/mercurial/help/internals/wireprotocol.txt
@@ -1923,6 +1923,67 @@
addition to the raw file data, it may contain a header defining metadata
such as copy information.
+filerevisionsslice
+------------------
+
+Obtain fulltext file revision data for a set of files on a specific revision.
+
+This command allows the caller to obtain the fulltext file revision data
+for 1 or more file paths, specified as a range between two paths.
+
+The command accepts the following arguments:
+
+node
+ (bytestring) Changeset revision whose file data to obtain.
+
+startpath
+ (bytestring) The tracked path at which file iteration should begin.
+
+endpath
+ (bytestring) The tracked path at which file iteration should end.
+
+This command effectively opens the manifest for the requested start
+revision, *seeks* to ``startpath`` and then starts iterating paths until
+it reaches ``endpath``. For each path encountered - including ``startpath``
+and ``endpath`` themselves - fulltext file revision data is obtained.
+
+The response bytestream starts with a CBOR map that defines the payload
+to follow. This map has the following bytestring keys:
+
+totalitems (optional)
+ (unsigned integer) Total number of files whose data will be returned.
+
+totalrevisionsize (optional)
+ (unsigned integer) Total size of all file revisions data, in bytes.
+
+Following the CBOR map header are a series of segments. Each segments consists
+of a CBOR map followed by a bytestring, possibly of indefinite length,
+containing the revision data. The CBOR map has the following bytestring keys:
+
+path
+ (bytestring) Tracked file path.
+
+node
+ (bytestring) File revision (what should appear in manifest).
+
+censored (optional)
+ (boolean) Indicates whether the revision data has been censored. If true,
+ ``size`` should be 0. An empty bytestring will follow this CBOR map.
+
+renamed (optional)
+ (array) If present, will be an array of 2 bytestrings defining the
+ copy/rename metadata for this revision. The first element is the path
+ the file was copied from. The second is its node revision.
+
+size
+ (unsigned integer) The size of the revision's fulltext data. The size is
+ not the count of bytes that follow this CBOR map, as the revision data
+ is CBOR encoded.
+
+Revision data may contain a header with metadata. So a consumer cannot stream
+the revision data to a file without first processing the stream looking
+for this header.
+
heads
-----
To: indygreg, #hg-reviewers
Cc: mercurial-devel
More information about the Mercurial-devel
mailing list