Further adventures in revlog compression

Bryan O'Sullivan bos at serpentine.com
Wed Jun 6 14:19:02 CDT 2012


I've spent the past couple of days digging deeper into the potential of
using lz4 as an alternative to zlib for compressing the contents of revlogs.

The RFC patch that I just posted is a current snapshot of what I have, so
that other motivated folks can give it a try. To try out lz4, first install
the Python lz4 library ("pip install lz4" or "easy_install lz4"), then "hg
clone --pull --config format.uselz4=1 my-repo my-repo-lz4".

OK, now that you can use lz4 yourself, here's the backstory.

The performance of zlib is a bottleneck for all operations that need to
read or write revisions in revlogs. The problem does not lie in the Python
zlib bindings, but in zlib itself, and I suspect that it's intrinsic to the
kind of compression that zlib does. If you use statprof or perf to profile
an update, clone (using the wire protocol), or pull, you'll find that
zlib's compression and decompression routines are consistently at or near
the top of the profile; it often takes up about 50% of all CPU time.

With the lz4 patch applied, the raw performance of "hg update" improves
dramatically.

   - In a hg repository using zlib, update from null to tip takes 0.884
   seconds. Using lz4, this drops to 0.67 seconds.
   - In a linux-2.6 repository, the same benchmark improves from 27 seconds
   to 22.

Of course this comes at a cost, namely disk space. lz4 offers two
compressors. The default is blazingly fast, 10 to 15 times faster than
zlib, but offers a lower compression ratio. The lz4 HC compressor is about
the same speed as the zlib compressor, and gets closer to its compression
ratio. (Interestingly, data compressed with HC decompresses about 10%
faster than with the regular compressor, I assume due to either cache or
memory bandwidth effects.)

   - A hg repo with zlib is 43MB. With lz4 HC, it's 53MB. With plain lz4,
   61MB.
   - A linux-2.6 repo with zlib is 2.6GB. With lz4 HC, it's 3.4GB. With
   plain lz4, 4.1GB. Most of the extra space consumed is due to the manifest
   getting larger with lz4.

lz4 only starts to make a perceptible difference once you've got a large
repo, at which point the larger revlogs generated by the fast compressor
become a potential source of memory pressure. As a result, I've chosen to
use the HC compressor in the patch. The difference in write performance is
very noticeable for bulk operations - it takes 9 minutes to clone --pull a
linux-2.6 tree with plain lz4, but 16 minutes with both zlib and lz4 HC.
Maybe the choice of compressor to use is worth offering as a configurable
knob.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://selenic.com/pipermail/mercurial-devel/attachments/20120606/1d4b648e/attachment.html>


More information about the Mercurial-devel mailing list