Experiments with LZMA compression of bundles

[Benoit Boissinot]

On Fri, May 20, 2011 at 12:20 PM, Sune Foldager <cryo at cyanite.org> wrote:
> On 2011-05-20 10:17, Benoit Boissinot wrote:
>> Note that the vast majority of bundle operation are over http/ssh, in
>> that case bzip2 won't be used, so the fair comparison is against zlib.
>
> It's a bit messy, really, as far as I can see:
>
> - hg bundle and hg push http:// writes/sends full bundles with proper
> compression.
>  For the http push case, a capability determines the supported types.
> - hg push ssh:// sends headerless bundles with no compression.
> - hg pull (both getbundle and changegroup(subset)) receive headerless
> bundles,
>  with zlib compression on top of it for http (and none for ssh).
>
> Is this for historic reasons? It seems unnecessarily complicated. Why
> wouldn't
> we always use headered bundles, say, and then have pull use HG10GZ; why wrap
> in a zlib stream and use uncompressed headerless bundles?

Historical reasons, we "forgot" to change that when we added the
compression capabilities for unbundle (which is only meaningful for
ssh, another screw up).
>
> Why does push (the unbundle remote command) send full, headered, bundles
> with
> arbitrary compression in the bundle, and no wrapper compression?
>
> Is this because pushes are often smaller than pulls? But that still doesn't
> address why we don't pull GZ bundles.

Historical again, http push was implemented later. Before the
wireprotocol unification we didn't spot those inconsistencies as
easily as we do now.

>> My guess is that the CPU usage on the server will show a much bigger
>> difference.
>
> Yes, for sure; lzma is a bit on the expensive side for compression. Hardware
> does get faster, though.
>
> Anyway, I am not proposing adding LZMA in at this time, or anything like
> that.
> It was just some experiments I thought might interest people.

Yes, experimenting is cool (for fast server side compression, with
better compression level than zlib, snappy/zippy would be interesting
too).

Benoit