Vendoring zstd

[Gregory Szorc]

> On Nov 15, 2016, at 06:31, Pierre-Yves David <pierre-yves.david at ens-lyon.org> wrote:
> 
> 
> 
>> On 11/10/2016 10:22 PM, Augie Fackler wrote:
>>> On Thu, Nov 10, 2016 at 10:07:32AM -0800, Gregory Szorc wrote:
>>> I plan to add support for zstd to Mercurial this release cycle. I don't
>>> think support itself is too controversial. (There will be bikeshedding over
>>> implementation details, of course.)
>>> 
>>> Since zstd is relatively new and not part of a Python distribution, we
>>> can't rely on its shared libraries being available on systems or even part
>>> of the system packager. In addition, the Python bindings to zstd rely on
>>> zstd APIs that require static linking to access. In addition, said zstd
>>> APIs are not guaranteed stable (which is why they aren't exported). So it
>>> is important for the Python bindings to be compiled against a known version
>>> of zstd. This is why the Python bindings ship the zstd sources and don't
>>> treat zstd as an external dependency. On top of that, the Python bindings
>>> are a bit fluid and targeting multiple versions from Mercurial could be
>>> painful until the API matures.
>>> 
>>> If we're going to have proper zstd support that "just works," I think we
>>> need to vendor the Python bindings (which includes the zstd sources) into
>>> the hg repo.
>>> 
>>> I have 2 questions:
>>> 
>>> 1) Is vendoring python-zstandard (
>>> https://github.com/indygreg/python-zstandard) (or at least the parts we
>>> need) acceptable?
>> 
>> It's fine with me. We might want to send a note out to the packaging
>> list giving them a heads up that:
>> 
>> 1) We're doing this
>> 2) We'd appreciate it if they didn't unbundle zstd, given the above
>> 
>> On our end, we should probably degrade gracefully if python-zstandard
>> isn't available, so that distros can unbundle and just not have the
>> dependency (at least until the APIs settle down).
> 
> I'm fine with that too and Augie extra step seems good.
> 
> To me, the main reason to vendor this is the API instability (the format itself is stable, right?).

The format should be stable.

> 
> Otherwise having Mercurial degrade gracefully if zst is not available seems like the best behavior for now (while zstd in Mercurial get mature). Later We could build package includig zstd (as we do for python2.7) for distribution that are missing it and just use the distribution package for the other (provided the API eventually stabilize). Maintaining zstd package is more of a distribution job and I would be happier if we do not have to handle security maintainance of zstd on the long run.

In the series I sent, there is not an option to disable zstd. I can do that as a follow-up enhancement to setup.py.

> 
>>> 2) Should I patchbomb the ~1 MB of sources or should the vendoring
>>> changeset(s) be submitted a different way?
>> 
>> I think for the monster import change, we should probably just have
>> you send a pull request.
> 
> +1
> 
> -- 
> Pierre-Yves David