zstd experimentations

Gregory Szorc gregory.szorc at gmail.com
Sat Sep 3 16:37:25 EDT 2016

https://github.com/indygreg/python-zstandard contains a proper, general use
C extension. I've implemented most of the C API, including support for
training and using dictionaries.

It is very alpha. I'm sure there are memory leaks from me not using the
Python C API properly (this is the first Python C extension I've written).
I'm sure it will crash. It also only runs on Python 3.5 at the moment. I'll
backport to 2.6 so we can experiment with this in Mercurial.

If you want a Mercurial clone of the repo, grab it from

On Thu, Sep 1, 2016 at 1:49 PM, Bryan O'Sullivan <bos at serpentine.com> wrote:

> It would be valuable to target a general-purpose package, if only because
> the existing Python zstd bindings (somewhere on github) are not good.
> You're right that zstd is a big win over zlib in every way.
> On Thu, Sep 1, 2016 at 12:01 AM, Gregory Szorc <gregory.szorc at gmail.com>
> wrote:
>> I got caught up in the zstd 1.0 release excitement and started writing a
>> proper Python C extension for it (one that uses the streaming API and
>> doesn't use custom framing like the existing python-zstd package does). The
>> result of what I was able to hack together in a few hours is available at
>> https://hg.mozilla.org/users/gszorc_mozilla.com/hg/rev/zstd (see the p1
>> changeset for the C code).
>> It compiles, but that's about it. The compress API basically no-ops (I
>> need to plug in iterator foo). The decompress API segfaults and the API
>> needs refined. It's also my first time writing a Python C extension, so I'm
>> sure I'm doing several things wrong. And I know the style doesn't match
>> Mercurial conventions. I'm not sure if I should strive for tailoring this
>> to Mercurial's usage or try to release it as a standalone package...
>> If anyone wants to take the code and run with it, go for it. I can stamp
>> another license on the code if someone asks.
>> I'd also like to invite a larger discussion around supporting zstd in the
>> official Mercurial distribution. It seems like an all-around better
>> compression format than zlib and supporting zstd seems like a relatively
>> easy way to gain significant performance and size efficiency wins on many
>> operations.
>> _______________________________________________
>> Mercurial-devel mailing list
>> Mercurial-devel at mercurial-scm.org
>> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160903/089c139d/attachment.html>

More information about the Mercurial-devel mailing list