D2836: wireproto: define permissions-based routing of HTTPv2 wire protocol

indygreg (Gregory Szorc) phabricator at mercurial-scm.org
Tue Mar 13 22:49:38 EDT 2018


indygreg updated this revision to Diff 7014.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D2836?vs=7010&id=7014

REVISION DETAIL
  https://phab.mercurial-scm.org/D2836

AFFECTED FILES
  mercurial/debugcommands.py
  mercurial/help/internals/wireprotocol.txt
  mercurial/wireprotoserver.py
  tests/test-http-api-httpv2.t

CHANGE DETAILS

diff --git a/tests/test-http-api-httpv2.t b/tests/test-http-api-httpv2.t
--- a/tests/test-http-api-httpv2.t
+++ b/tests/test-http-api-httpv2.t
@@ -1,3 +1,5 @@
+  $ HTTPV2=exp-http-v2-0001
+
   $ send() {
   >   hg --verbose debugwireproto --peer raw http://$LOCALIP:$HGPORT/
   > }
@@ -13,7 +15,7 @@
 HTTP v2 protocol not enabled by default
 
   $ send << EOF
-  > httprequest GET api/exp-http-v2-0001
+  > httprequest GET api/$HTTPV2
   >     user-agent: test
   > EOF
   using raw connection to peer
@@ -43,14 +45,14 @@
   $ hg -R server serve -p $HGPORT -d --pid-file hg.pid
   $ cat hg.pid > $DAEMON_PIDS
 
-Requests simply echo their path (for now)
+Request to read-only command works out of the box
 
   $ send << EOF
-  > httprequest GET api/exp-http-v2-0001/path1/path2
+  > httprequest GET api/$HTTPV2/ro/capabilities
   >     user-agent: test
   > EOF
   using raw connection to peer
-  s>     GET /api/exp-http-v2-0001/path1/path2 HTTP/1.1\r\n
+  s>     GET /api/exp-http-v2-0001/ro/capabilities HTTP/1.1\r\n
   s>     Accept-Encoding: identity\r\n
   s>     user-agent: test\r\n
   s>     host: $LOCALIP:$HGPORT\r\n (glob)
@@ -60,6 +62,178 @@
   s>     Server: testing stub value\r\n
   s>     Date: $HTTP_DATE$\r\n
   s>     Content-Type: text/plain\r\n
-  s>     Content-Length: 12\r\n
+  s>     Content-Length: 16\r\n
+  s>     \r\n
+  s>     ro/capabilities\n
+
+Request to unknown command yields 404
+
+  $ send << EOF
+  > httprequest GET api/$HTTPV2/ro/badcommand
+  >     user-agent: test
+  > EOF
+  using raw connection to peer
+  s>     GET /api/exp-http-v2-0001/ro/badcommand HTTP/1.1\r\n
+  s>     Accept-Encoding: identity\r\n
+  s>     user-agent: test\r\n
+  s>     host: $LOCALIP:$HGPORT\r\n (glob)
+  s>     \r\n
+  s> makefile('rb', None)
+  s>     HTTP/1.1 404 Not Found\r\n
+  s>     Server: testing stub value\r\n
+  s>     Date: $HTTP_DATE$\r\n
+  s>     Content-Type: text/plain\r\n
+  s>     Content-Length: 42\r\n
+  s>     \r\n
+  s>     unknown wire protocol command: badcommand\n
+
+Request to read-write command fails because server is read-only by default
+
+GET to read-write request not allowed
+
+  $ send << EOF
+  > httprequest GET api/$HTTPV2/rw/capabilities
+  >     user-agent: test
+  > EOF
+  using raw connection to peer
+  s>     GET /api/exp-http-v2-0001/rw/capabilities HTTP/1.1\r\n
+  s>     Accept-Encoding: identity\r\n
+  s>     user-agent: test\r\n
+  s>     host: $LOCALIP:$HGPORT\r\n (glob)
+  s>     \r\n
+  s> makefile('rb', None)
+  s>     HTTP/1.1 405 push requires POST request\r\n
+  s>     Server: testing stub value\r\n
+  s>     Date: $HTTP_DATE$\r\n
+  s>     Content-Length: 17\r\n
+  s>     \r\n
+  s>     permission denied
+
+Even for unknown commands
+
+  $ send << EOF
+  > httprequest GET api/$HTTPV2/rw/badcommand
+  >     user-agent: test
+  > EOF
+  using raw connection to peer
+  s>     GET /api/exp-http-v2-0001/rw/badcommand HTTP/1.1\r\n
+  s>     Accept-Encoding: identity\r\n
+  s>     user-agent: test\r\n
+  s>     host: $LOCALIP:$HGPORT\r\n (glob)
+  s>     \r\n
+  s> makefile('rb', None)
+  s>     HTTP/1.1 405 push requires POST request\r\n
+  s>     Server: testing stub value\r\n
+  s>     Date: $HTTP_DATE$\r\n
+  s>     Content-Length: 17\r\n
+  s>     \r\n
+  s>     permission denied
+
+SSL required by default
+
+  $ send << EOF
+  > httprequest POST api/$HTTPV2/rw/capabilities
+  >     user-agent: test
+  > EOF
+  using raw connection to peer
+  s>     POST /api/exp-http-v2-0001/rw/capabilities HTTP/1.1\r\n
+  s>     Accept-Encoding: identity\r\n
+  s>     user-agent: test\r\n
+  s>     host: $LOCALIP:$HGPORT\r\n (glob)
+  s>     \r\n
+  s> makefile('rb', None)
+  s>     HTTP/1.1 403 ssl required\r\n
+  s>     Server: testing stub value\r\n
+  s>     Date: $HTTP_DATE$\r\n
+  s>     Content-Length: 17\r\n
   s>     \r\n
-  s>     path1/path2\n
+  s>     permission denied
+
+Restart server to allow non-ssl read-write operations
+
+  $ killdaemons.py
+  $ cat > server/.hg/hgrc << EOF
+  > [experimental]
+  > web.apiserver = true
+  > web.api.http-v2 = true
+  > [web]
+  > push_ssl = false
+  > EOF
+
+  $ hg -R server serve -p $HGPORT -d --pid-file hg.pid
+  $ cat hg.pid > $DAEMON_PIDS
+
+Server insists on POST for read-write commands
+
+  $ send << EOF
+  > httprequest GET api/$HTTPV2/rw/capabilities
+  >     user-agent: test
+  > EOF
+  using raw connection to peer
+  s>     GET /api/exp-http-v2-0001/rw/capabilities HTTP/1.1\r\n
+  s>     Accept-Encoding: identity\r\n
+  s>     user-agent: test\r\n
+  s>     host: $LOCALIP:$HGPORT\r\n (glob)
+  s>     \r\n
+  s> makefile('rb', None)
+  s>     HTTP/1.1 405 push requires POST request\r\n
+  s>     Server: testing stub value\r\n
+  s>     Date: $HTTP_DATE$\r\n
+  s>     Content-Length: 17\r\n
+  s>     \r\n
+  s>     permission denied
+
+  $ killdaemons.py
+  $ cat > server/.hg/hgrc << EOF
+  > [experimental]
+  > web.apiserver = true
+  > web.api.http-v2 = true
+  > [web]
+  > push_ssl = false
+  > allow-push = *
+  > EOF
+
+  $ hg -R server serve -p $HGPORT -d --pid-file hg.pid
+  $ cat hg.pid > $DAEMON_PIDS
+
+Authorized request for valid read-write command works
+
+  $ send << EOF
+  > httprequest POST api/$HTTPV2/rw/capabilities
+  >     user-agent: test
+  > EOF
+  using raw connection to peer
+  s>     POST /api/exp-http-v2-0001/rw/capabilities HTTP/1.1\r\n
+  s>     Accept-Encoding: identity\r\n
+  s>     user-agent: test\r\n
+  s>     host: $LOCALIP:$HGPORT\r\n (glob)
+  s>     \r\n
+  s> makefile('rb', None)
+  s>     HTTP/1.1 200 OK\r\n
+  s>     Server: testing stub value\r\n
+  s>     Date: $HTTP_DATE$\r\n
+  s>     Content-Type: text/plain\r\n
+  s>     Content-Length: 16\r\n
+  s>     \r\n
+  s>     rw/capabilities\n
+
+Authorized request for unknown command is rejected
+
+  $ send << EOF
+  > httprequest POST api/$HTTPV2/rw/badcommand
+  >     user-agent: test
+  > EOF
+  using raw connection to peer
+  s>     POST /api/exp-http-v2-0001/rw/badcommand HTTP/1.1\r\n
+  s>     Accept-Encoding: identity\r\n
+  s>     user-agent: test\r\n
+  s>     host: $LOCALIP:$HGPORT\r\n (glob)
+  s>     \r\n
+  s> makefile('rb', None)
+  s>     HTTP/1.1 404 Not Found\r\n
+  s>     Server: testing stub value\r\n
+  s>     Date: $HTTP_DATE$\r\n
+  s>     Content-Type: text/plain\r\n
+  s>     Content-Length: 42\r\n
+  s>     \r\n
+  s>     unknown wire protocol command: badcommand\n
diff --git a/mercurial/wireprotoserver.py b/mercurial/wireprotoserver.py
--- a/mercurial/wireprotoserver.py
+++ b/mercurial/wireprotoserver.py
@@ -272,6 +272,64 @@
                                    req.dispatchparts[2:])
 
 def _handlehttpv2request(rctx, req, res, checkperm, urlparts):
+    from .hgweb import common as hgwebcommon
+
+    # URL space looks like: <permissions>/<command>, where <permission> can
+    # be ``ro`` or ``rw`` to signal read-only or read-write, respectively.
+
+    # Root URL does nothing meaningful... yet.
+    if not urlparts:
+        res.status = b'200 OK'
+        res.headers[b'Content-Type'] = b'text/plain'
+        res.setbodybytes(_('HTTP version 2 API handler'))
+        return
+
+    if len(urlparts) == 1:
+        res.status = b'404 Not Found'
+        res.headers[b'Content-Type'] = b'text/plain'
+        res.setbodybytes(_('do not know how to process %s\n') %
+                         req.dispatchpath)
+        return
+
+    permission, command = urlparts[0:2]
+
+    if permission not in (b'ro', b'rw'):
+        res.status = b'404 Not Found'
+        res.headers[b'Content-Type'] = b'text/plain'
+        res.setbodybytes(_('unknown permission: %s') % permission)
+        return
+
+    # At some point we'll want to use our own API instead of recycling the
+    # behavior of version 1 of the wire protocol...
+    # TODO return reasonable responses - not responses that overload the
+    # HTTP status line message for error reporting.
+    try:
+        checkperm(rctx, req, 'pull' if permission == b'ro' else 'push')
+    except hgwebcommon.ErrorResponse as e:
+        res.status = hgwebcommon.statusmessage(e.code, pycompat.bytestr(e))
+        for k, v in e.headers:
+            res.headers[k] = v
+        res.setbodybytes('permission denied')
+        return
+
+    if command not in wireproto.commands:
+        res.status = b'404 Not Found'
+        res.headers[b'Content-Type'] = b'text/plain'
+        res.setbodybytes(_('unknown wire protocol command: %s\n') % command)
+        return
+
+    repo = rctx.repo
+    ui = repo.ui
+
+    proto = httpv2protocolhandler(req, ui)
+
+    if not wireproto.commands.commandavailable(command, proto):
+        res.status = b'404 Not Found'
+        res.headers[b'Content-Type'] = b'text/plain'
+        res.setbodybytes(_('invalid wire protocol command: %s') % command)
+        return
+
+    # We don't do anything meaningful yet.
     res.status = b'200 OK'
     res.headers[b'Content-Type'] = b'text/plain'
     res.setbodybytes(b'/'.join(urlparts) + b'\n')
@@ -284,6 +342,34 @@
     },
 }
 
+class httpv2protocolhandler(wireprototypes.baseprotocolhandler):
+    def __init__(self, req, ui):
+        self._req = req
+        self._ui = ui
+
+    @property
+    def name(self):
+        return HTTPV2
+
+    def getargs(self, args):
+        raise NotImplementedError
+
+    def forwardpayload(self, fp):
+        raise NotImplementedError
+
+    @contextlib.contextmanager
+    def mayberedirectstdio(self):
+        raise NotImplementedError
+
+    def client(self):
+        raise NotImplementedError
+
+    def addcapabilities(self, repo, caps):
+        raise NotImplementedError
+
+    def checkperm(self, perm):
+        raise NotImplementedError
+
 def _httpresponsetype(ui, req, prefer_uncompressed):
     """Determine the appropriate response type and compression settings.
 
diff --git a/mercurial/help/internals/wireprotocol.txt b/mercurial/help/internals/wireprotocol.txt
--- a/mercurial/help/internals/wireprotocol.txt
+++ b/mercurial/help/internals/wireprotocol.txt
@@ -144,6 +144,46 @@
 ``application/mercurial-0.*`` media type and the HTTP response is typically
 using *chunked transfer* (``Transfer-Encoding: chunked``).
 
+HTTP Version 2 Transport
+------------------------
+
+**Experimental - feature under active development**
+
+Version 2 of the HTTP protocol is exposed under the ``/api/*`` URL space.
+It's final API name is not yet formalized.
+
+Commands are triggered by sending HTTP requests against URLs of the
+form ``<permission>/<command>``, where ``<permission>`` is ``ro`` or
+``rw``, meaning read-only and read-write, respectively and ``<command>``
+is a named wire protocol command.
+
+Commands that modify repository state in meaningful ways MUST NOT be
+exposed under the ``ro`` URL prefix. All available commands MUST be
+available under the ``rw`` URL prefix.
+
+Server adminstrators MAY implement blanket HTTP authentication keyed
+off the URL prefix. For example, a server may require authentication
+for all ``rw/*`` URLs and let unauthenticated requests to ``ro/*``
+URL proceed. A server MAY issue an HTTP 401, 403, or 407 response
+in accordance with RFC 7235. Clients SHOULD recognize the HTTP Basic
+(RFC 7617) and Digest (RFC 7616) authentication schemes. Clients SHOULD
+make an attempt to recognize unknown schemes using the
+``WWW-Authenticate`` response header on a 401 response, as defined by
+RFC 7235.
+
+Read-only commands are accessible under ``rw/*`` URLs so clients can
+signal the intent of the operation very early in the connection
+lifecycle. For example, a ``push`` operation - which consists of
+various read-only commands mixed with at least one read-write command -
+can perform all commands against ``rw/*`` URLs so that any server-side
+authentication requirements are discovered upon attempting the first
+command - not potentially several commands into the exchange. This
+allows clients to fail faster or prompt for credentials as soon as the
+exchange takes place. This provides a better end-user experience.
+
+Requests to unknown commands or URLS result in an HTTP 404.
+TODO formally define response type, how error is communicated, etc.
+
 SSH Protocol
 ============
 
diff --git a/mercurial/debugcommands.py b/mercurial/debugcommands.py
--- a/mercurial/debugcommands.py
+++ b/mercurial/debugcommands.py
@@ -2956,6 +2956,11 @@
             url = path + httppath
             req = urlmod.urlreq.request(pycompat.strurl(url), body, headers)
 
+            # urllib.Request insists on using has_data() as a proxy for
+            # determining the request method. Override that to use our
+            # explicitly requested method.
+            req.get_method = lambda: method
+
             try:
                 opener.open(req).read()
             except util.urlerr.urlerror as e:



To: indygreg, #hg-reviewers
Cc: mercurial-devel


More information about the Mercurial-devel mailing list