[PATCH 7 of 9 RFC] pushkey: support for encoding and decoding raw listkeys dicts

[Gregory Szorc]

On Tue, Aug 16, 2016 at 6:57 AM, Yuya Nishihara <yuya at tcha.org> wrote:

> On Sun, 14 Aug 2016 14:10:06 -0700, Gregory Szorc wrote:
> > # HG changeset patch
> > # User Gregory Szorc <gregory.szorc at gmail.com>
> > # Date 1471207500 25200
> > #      Sun Aug 14 13:45:00 2016 -0700
> > # Node ID eb2bc1ac7869ad255965d16004524a95cea83c9d
> > # Parent  1fe812eb8b9e79d1182c4a6593e7ce8fa2938264
> > pushkey: support for encoding and decoding raw listkeys dicts
>
> > +def encodekeysraw(keys):
> > +    """Encode pushkey namespace keys using a binary encoding.
> > +
> > +    The response consists of framed data packets of the form:
> > +
> > +        <size> <data>
> > +
> > +    Where the ``size`` is a little endian 32-bit integer.
> > +
> > +    Data is emitted in pairs of frames where the first frame is the key
> > +    name and the second frame is the value.
> > +
> > +    A frame with size 0 indicates end of stream.
> > +    """
> > +    s = struct.struct('<I')
> > +
> > +    chunks = []
> > +    for k, v in keys:
> > +        assert not isinstance(k, encoding.localstr)
> > +        assert not isinstance(v, encoding.localstr)
> > +
> > +        chunks.append(s.pack(len(k)))
> > +        chunks.append(k)
> > +        chunks.append(s.pack(len(v)))
> > +        chunks.append(v)
>
> I heard we should stick to big endian. The cost of byte-order conversion
> should be pretty cheap.
>
> And if we're trying to reduce the payload size, it might be too large to
> add 4-byte length field for each key/value pair.
>

Yeah. Keys can almost certainly be 16-bit. Values I'm less sure of. I could
see us going over 64k for things like obsolescence markers. Although, there
is something to be said about limiting sizes. We shouldn't be having
listkeys transfer large payloads because it is intrinsically a buffered
command/protocol and large payloads should be streamed for performance
reasons.

If we really cared about sizes, we could introduce a "varint." I'm pretty
sure I've heard the Googler's propose this before.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160816/fb7ca5ef/attachment.html>