Rust extensions: the next step

[Gregory Szorc]

> On Oct 17, 2018, at 18:45, Georges Racinet <gracinet at anybox.fr> wrote:
> 
> Hi all,
> 
> first, many thanks for the Stockholm sprint, it was my first interaction
> with the Mercurial community, and it's been very welcoming to me.
> 
> I've been pursuing some experiments I started then to convert the Rust
> bindings I've done in the patch series about ancestry iteration (now
> landed) to a proper Python extension, using the cpython crate and Python
> capsules. In short, it works.
> 
> Early benchmarking shows that it's a few percent slower than the direct
> bindings through C code, which I think is acceptable compared to the
> other benefits (clearer integration, easier to generalise, no C code at
> all).
> 
> The end result is a unique shared library importable as
> 'mercurial.rustext', which is itself made of several submodules, ie, one
> can do:
> 
>    from mercurial.rustext.ancestor import AncestorsIterator

This all sounds very reasonable to me.

One open item is full PyPy/cffi support. Ideally we’d only write the native code interface once. But I think that means cffi everywhere and last I looked, CPython into cffi was a bit slower compared to native extensions. I’m willing to ignore cffi support for now (PyPy can use pure Python and rely on JIT for faster execution). Maybe something like milksnake can help us here? But I’m content with using the cpython crate to maintain a Rust-based extension: that’s little different from what we do today and we shouldn’t let perfect be the enemy of good.

Something else we may want to consider is a single Python module exposing the Rust code instead of N. Rust’s more aggressive cross function compilation optimization could result in better performance if everything is linked/exposed in a single shared library/module/extension. Maybe this is what you are proposing? It is unclear if Rust code is linked into the Python extension or loaded from a shared shared library.

> 
> It will take me some more time, though, to get that experiment into a
> reviewable state (have to switch soon to other, unrelated, works) and
> we're too close to the freeze anyway, but if someone wants to see it, I
> can share it right away.
> 
> Also, I could summarize some of these thoughts on the Oxidation wiki
> page. Greg, are you okay with that ?

Yes, please update the Oxidation wiki! I’ve been meaning to update it with results of discussions at the sprint. I’ve just been busy trying to finish my patches for 4.8...

> 
> Regards,
> 
> -- 
> Georges Racinet
> Anybox SAS, http://anybox.fr
> Téléphone: +33 6 51 32 07 27
> GPG: B59E 22AB B842 CAED 77F7 7A7F C34F A519 33AB 0A35, sur serveurs publics
> 
> 
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel