[PATCH RFC] wireproto: add support for batching of some commands

Peter Arrenbrecht peter.arrenbrecht at gmail.com
Wed Mar 16 11:10:28 CDT 2011


On Mon, Mar 14, 2011 at 7:44 PM, Matt Mackall <mpm at selenic.com> wrote:
> On Mon, 2011-03-14 at 10:07 +0100, Peter Arrenbrecht wrote:
>> # HG changeset patch
>> # User Peter Arrenbrecht <peter.arrenbrecht at gmail.com>
>> # Date 1300093067 -3600
>> wireproto: add support for batching of some commands
>>
>> Allows clients to send batches of server commands in one roundtrip. An
>> example is provided in the contained test.
>>
>> This is desirable for the new discovery protocol. For example, in a later
>> patch we will batch the request for the server's heads with the one where
>> the server checks if it knows the client's heads, eliminating one roundtrip
>> out of 4 for a pull where local is a subset of remote.
>
> It'd be really nice if you explained how it worked. There's a lot here I
> don't understand.
>
> I can't find the part where the batched command automatically falls back
> to serial commands so that client code doesn't have to implement
> fallback on its own.

OK, so. Attached is a sketch of something more comprehensive and
usable than my last patch, but also more complex. I kept it isolated
at this stage, but the goal is, of course, to use it mainly for
local/wirerepository. Right now I am interested in feedback on the
approach before spending time trying to make wireproto and discovery
use it. (Maybe you all think this is going way too far.)

They key is that we need to separate the encoding of commands and
arguments and the decoding of results a bit so they can be recombined
in different ways. For direct calls, they are simply in sequence:
encode/submit/decode. For batching, this obviously has to become
{encode}/submit/{decode} where we first encode all commands in the
batch, then submit the batch, then decode all results.

The code uses a bit of __getattr__ and __call__ magic to automatically
mirror the main/proxy class's API into the API for batches of calls,
and to minimize the code needed for method proxies. it also uses a
minimalistic version of futures to hold the results of batched calls.
Here's a sample usage (where "it" is the object under test - either
local or proxied):

    # Direct call to base method shared between client and server.
    print it.hello()

    # Direct call to possibly proxied method.
    # You currently have to use named arguments when proxies are synthesized.
    print it.foo(one="Un", two="Deux")

    # Here's a call with an explicit proxy supporting positional arguments.
    print it.bar("Eins", "Zwei")

    # Batched call to a couple of proxied methods.
    # You cannot mix proxied and shared methods at the moment.
    batch = it.batch()
    # The calls return futures (single-element lists) to eventually
hold the results.
    fooref = batch.foo(one="One", two="Two")
    # Within batches you always have to use named arguments.
    barref = batch.bar(eins="Eins", zwei="Zwei")
    greetref = batch.greet(name="John Smith")
    bar2ref = batch.bar(eins="Uno", zwei="Due")
    # Only now are all the calls executed in sequence, with as little roundtrips
    # as possible.
    batch.submit()
    # After the call to submit, the futures actually contain values.
    print fooref[0]
    print barref[0]
    print greetref[0]
    print bar2ref[0]

A local execution says:

Ready.
Un and Deux
Eins und Zwei
One and Two
Eins und Zwei
Hello, John Smith
Uno und Due

And a proxied one (note that params are mangled in the requests to
test encode/decode; also note that the batch is automatically split
because greet is not batchable):

Ready.
REQ: foo?one=Vo&two=Efvy
  -> Vo!boe!Efvy
Un and Deux
REQ: bar?eins=Fjot&zwei=[xfj
  -> Fjot!voe![xfj
Eins und Zwei
REQ: batch?cmds=foo:one=Pof,two=Uxp;bar:eins=Fjot,zwei=[xfj
  -> Pof!boe!Uxp;Fjot!voe![xfj
REQ: greet?name=Kpio!Tnjui
  -> Ifmmp-!Kpio!Tnjui
REQ: bar?eins=Vop&zwei=Evf
  -> Vop!voe!Evf
One and Two
Eins und Zwei
Hello, John Smith
Uno und Due

Adding batching support to the local implementation is dead simple:

# equivalent of localrepo.localrepository
class TestLocal(TestBase):
    def foo(self, one=None, two=None):
        return "%s and %s" % (one, two,)
    def bar(self, eins, zwei):
        return "%s und %s" % (eins, zwei,)
    def greet(self, name=None):
        return "Hello, %s" % name
    def batch(self):
        '''Support for local batching.'''
        return Batch(self, LOCAL)

For the remote proxy it is a bit more work, but still pretty straightforward:

# equivalent of wireproto.wirerepository
class TestProxy(TestBase, TestProxySupport):
    def __init__(self, server):
        TestProxySupport.__init__(self, server)

    # foo has encode/decode methods so it supports batching. The actual call to
    # foo is synthesized.
    def encode_foo(self, one=None, two=None):
        return [('one', mangle(one),), ('two', mangle(two),)]
    def decode_foo(self, res):
        return unmangle(res)

    # Bar is not synthesized, but also supports batching. This way we
can support
    # positional arguments. Maybe we can improve this by parsing the declaration
    # of encode_bar, which might then also support positional args in batches.
    def bar(self, eins, zwei):
        return ProxyCall(self, 'bar')(eins=eins, zwei=zwei)
    def encode_bar(self, eins=None, zwei=None):
        return [('eins', mangle(eins),), ('zwei', mangle(zwei),)]
    def decode_bar(self, res):
        return unmangle(res)

    # greet is coded directly. It therefore does not support batching. If it
    # does appear in a batch, the batch is split around greet, and the call to
    # greet is done in its own roundtrip.
    def greet(self, name=None):
        return unmangle(self._submitone('greet', [('name', mangle(name),)]))

Behind this is a bit of code that does the encoding and - in a real
implementation - would do wire transmission.

Thoughts?
-parren
-------------- next part --------------
A non-text attachment was scrubbed...
Name: batching.py
Type: text/x-python
Size: 9934 bytes
Desc: not available
URL: <http://selenic.com/pipermail/mercurial-devel/attachments/20110316/d7c022e6/attachment.py>


More information about the Mercurial-devel mailing list