[PATCH RFC] RFC: allow optional C++ 11 extensions with pybind11 for performance code

Tue Feb 9 17:32:48 EST 2016

El martes, 9 de febrero de 2016, Matthew Turk <matthewturk at gmail.com>
escribió:

> Hi Augie,
>
> On Tue, Feb 9, 2016 at 2:01 PM, Augie Fackler <raf at durin42.com
> <javascript:_e(%7B%7D,'cvml','raf at durin42.com');>> wrote:
>
>> On Mon, Feb 08, 2016 at 03:55:59PM -0800, Sean Farley wrote:
>> >
>> > timeless <timeless at gmail.com
>> <javascript:_e(%7B%7D,'cvml','timeless at gmail.com');>> writes:
>> >
>> > > Sean Farley <sean at farley.io
>> <javascript:_e(%7B%7D,'cvml','sean at farley.io');>> wrote:
>> > >> Yes, this is precisely the experience I have with C++ and libc++
>> (yuck
>> > >> and gross incompatibility with libstdc++). It's not just Solaris,
>> > >> either. Try anything that is not {Linux, Windows, Mac} / {gcc, clang}
>> > >> and you'll see all kinds of corner cases crop up.
>> > >
>> > > There are plenty of painful corner cases on Windows too.
>> > > And if you try hard enough you can get plenty of annoying corner
>> cases on Linux.
>> > >
>> > > Would a CFront style approach be good enough? Write in limited C++,
>> > > use llvm to convert it to C, commit the C++ as source and the
>> > > generated C and have the build system use the generated C?
>> >
>> > I'd rather not go down this route. It's a little too close to switching
>> > languages completely.
>> >
>> > By writing in C++ we are making it harder for us in the future to use
>> > pypy. I think we can all agree that writing Python C is annoying
>> > (opposed to just straight C). One way forward could be:
>> >
>> > - write standard C, creating a shared library
>> > - use cffi for binding to that
>> >
>>
>> Writing standard C is actually /worse/ than writing Python C. You have
>> even fewer nice things to work with (no map type out of the box, but
>> in Python C you can at least use PyDict). Part of the reason C++
>> appeals is precisely because it's a widely-available language with
>> better memory management tools (like smart pointers) and richer standard
>> tools available (STL collections come to mind).
>>
>> I don't dispute the idea that it'll make some things harder. I *am*
>> getting tired of the same old "HPC people can't get their shit
>> together" song and dance being the reason I can't have any nice things
>> and have to write a language that's fundamentally stuck in the 70s.
>>
>>
> On IRC I mentioned this, but I wanted to bring it up here as well.  I work
> on a python project that gets deployed on HPC clusters quite a lot, and we
> have long been resistant to C++ in the codebase.  The reasons for this have
> been similar to what Sean mentions (although I have not personally done
> much with BG machines) and in particular are often related to things like
> running on reduced OS nodes that don't have shared library loaders, etc etc.
>
> This has in the past led to needing to do any number of hoop-jumping to
> statically link a bunch of Python objects into a massive executable,
> mangling zipimport, and so on.  Lately though, we have (in our project,
> which may *not* translate elsewhere) found that these have not been issues;
> for the most part, deployments are now on machines that don't restrict
> nodes having shared libraries, things are slightly more up to date, etc
> etc.  C++11 support can be a little weak in places, but I don't have the
> numbers to back it up.  Statically linking can be really tricky with mixed
> codebases that sometimes share different compiler toolchains (i.e., mixing
> gcc with icc), which has caused us headaches previously.
>
> But, as I mentioned to Sean on IRC, in our project we have opened up a
> little bit to more C++ in the code base.  The machines we're finding are
> now much more up to date on compiler toolchains and we're not really
> needing to static link anymore.  For the use cases hg is used in, typically
> individuals will be running on "head" nodes which are usually much more
> feature rich anyway.
>
> The biggest issues I have *personally* seen with mercurial on HPC centers
> in the last year or two have been completely unrelated to mercurial.  These
> come down to:
>
>  * Old, out of date OpenSSL libraries or python installations
>  * Bad filesystem performance for metadata lookups
>
> I'll hold back from outright disagreeing with Sean (especially since IIRC
> he's more experienced in the DOE HPC side of things, and I'm more
> experienced in the NSF HPC side) but I don't think that from *my*
> perspective C++ embedded in a Python project would be a big barrier to
> adoption of hg on the HPC machines I use.  Although, come to think of it, I
> use matplotlib on *every* machine I work on, and it has a large C++
> component embedded in a Python project, and it has only been a problem when
> running on compute nodes, which hg probably doesn't need to do.
>
> -Matt
>
>
>> Put another way: I'm totally open to having some way to do cleaner
>> programming in C, but as it stands the tools we're able to use there
>> are disastrously difficult and error-prone.
>>
>>
>> >
>> > The benefit is that we can use our C code for pypy. The downside is that
>> > we introduce a cffi dependency.
>> >
>>
>> http://doc.pypy.org/en/release-1.9/cppyy.html is a thing for pypy. I
>> can't speak to its overall value or usabilty, but C++ does not
>> preclude use of pypy in the future.
>
>
Have you guys considered using some language that transpiles to C instead
of using C++?

I've recently been playing around with nim (http://nim-lang.org), which
compiles to C but has a much nicer syntax (it feels like a statically typed
Python) and a better standard library. It was surprisingly productive and
the resulting executable was as fast as I could have expected from a
program written in raw C or C++.

I'm not saying that using a niche language such as nim would be a good idea
for the mercurial project (nim has not hit version 1.0 yet, although it
will soon), but perhaps there are other ways to generate C code that could
be used (e.g. Cython, etc)?

Cheers,

Angel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160209/13e9f7ad/attachment.html>