Native support for lz4?

[Gregory Szorc]

On Fri, Aug 5, 2016 at 10:48 AM, Gregory Szorc <gregory.szorc at gmail.com>
wrote:

> Facebook introduced an lz4revlog extension a while ago. I think lz4 has
> some compelling performance advantages over zlib for revlog storage and
> wire protocol compression.
>
> I'd like to start a discussion about bundling the lz4 C implementation as
> part of the Mercurial distribution and supporting lz4 for revlogs and wire
> protocol compression out of the box.
>
> I'm not proposing requiring lz4 or making lz4 the default. I mostly care
> about making lz4 accessible to more users. (The 3rd party lz4revlog
> extension is difficult to use because you need a separate Python package
> providing lz4 support. Plus, lz4revlog isn't using the proper lz4 framing
> encoding and I'm hesitant to recommend its use because of this.)
>
> I'd also entertain scope bloating the conversation to including other
> compression formats. Once you support 2, you need to support N, right? I've
> been taking an interest in zstd and I'd be curious if Facebook, others have
> any plans to add support to Mercurial.
>

If anyone wants to play around, I hacked together lz4 revlog support. Code
is in the "lz4" bookmark at
https://hg.mozilla.org/users/gszorc_mozilla.com/hg/rev/lz4

If you build that locally, run `hg --config format.uselz4=true init repo`
to create a new repo that will use (framed) lz4 compression on revlogs.

I stopped short of adding lz4 support elsewhere, such as to bundle
compression and the wire protocol.

lz4 C code and a Python extension from
https://github.com/Iotic-Labs/py-lz4framed is vendored. We may want to
write our own minimal C extension. We'll also likely need to write cffi
glue for PyPy. Whether we write a simple C shim into the lz4 API or just
call the lz4 API directly remains to be seen (lz4's API seems simple enough
to call directly, but a wrapping layer exposing only the functionality we
need could be useful).

Resulting revlogs are slightly larger than the existing lz4revlog extension
(presumably due to presence of frames).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160807/1ae06560/attachment.html>