[PATCH RFC] RFC: allow optional C++ 11 extensions with pybind11 for performance code

[Sean Farley]

Durham Goode <durham at fb.com> writes:

> On 2/8/16 1:53 PM, Sean Farley wrote:
>> Laurent Charignon <lcharignon at fb.com> writes:
>>
>>> On 2/8/16, 1:33 PM, "Sean Farley" <sean at farley.io> wrote:
>>>
>>>> Laurent Charignon <lcharignon at fb.com> writes:
>>>>
>>>>> # HG changeset patch
>>>>> # User Laurent Charignon <lcharignon at fb.com>
>>>>> # Date 1454965979 28800
>>>>> #      Mon Feb 08 13:12:59 2016 -0800
>>>>> # Branch stable
>>>>> # Node ID 3c1ae9c7e93e8556fa7470919108a02bd989d040
>>>>> # Parent  61f4d59e9a0be4e25c1aa016db1a80a540a9d337
>>>>> RFC: allow optional C++ 11 extensions with pybind11 for performance code
>>>> I am very much against this as I am also against writing code in rust /
>>>> go / etc. Mercurial is a *system tool*. When cloning code on IBM
>>>> BlueGene it is annoying as hell to have random dependencies sneak in
>>>> like this. For reference, IBM BlueGene has had problem with codes that
>>>> use C++.
>>> I understand your concern but that is not the case, it is not a random
>>> dependency.
>>> Like our C extensions, the C++ extensions, would be *optional*.
>>> If you are not able to compile them on some system, mercurial should still
>>> work without them.
>> It would still be crippled (performance-wise). In academia (and HPC)
>> this was very embarrassing. Git could be compiled fast on the same
>> machine and was not crippled in the same way.
>>
>>>> C is much, much, much more portable. In fact, I would like to go even
>>>> further and break out the C code we have into a shared library (for cffi
>>>> sweetness).
>>> It is true that C is more portable.
>>> However, it is much more difficult (at least for me) to implement large
>>> features like lazy manifest or tree manifest in C compared to rust / go /
>>> C++.
>>> How do you think we should proceed?
>>> Do you suggest finding a C implementation of whatever
>>> data-structure/feature that would work for us and include them in the
>>> project?
>> Why not? I don't C is *that* hard. My usual advice is to prototype in
>> your favorite language of Python / rust / go / D / whatever and then
>> convert it to C for portability and performance.
> I think you underestimate the barrier of entry that C puts in the way.  
> There's plenty of places that would benefit from native fast paths 
> (revlog parsing, revset handling, etc), and a large reason we haven't 
> attempted to do it is because the developer-time vs benefit ratio for C 
> is so poor.

I think you mean Python C. Writing in standalone is a much different
experience IMHO. While C has a higher barrier than, say, Python, C++ is
not the direction you want to go.

> As for supporting super computers and solaris, if the current Mercurial 
> performance has been adequate for those edge cases for 10 years, I think 
> they will be fine with the existing perf for a while longer.

This is a false statement. Solaris / super computers / esoteric hardware
are not currently suffering *any* penalty because the fast paths today
in C. You are jeopardizing that with a move to C++.

>  I don't 
> think we should hinder development for the 99% platforms (linux, osx, 
> windows) just so we can get those same perf improvements on edge cases.  
> Hell, most repos are small enough that they won't even need the perf 
> improvements we're proposing using c++ for.

You and Google are too biased for me to believe "too small".

> That said, it's totally possible that C++ isn't an appropriate cross 
> platform solution for even linux+osx+windows.  I'm relying on the 
> experience of members of the community to determine the sanity of this idea.

If you have never had to support cross-platform C++, or even libc++ vs
libstdc++, then I envy you. It is a headache no on should have to deal
with.

As I mentioned before, why not try cffi with standard C? This route will
open the door for pypy as well as a portable shared library.