[PATCH RFC] RFC: allow optional C++ 11 extensions with pybind11 for performance code

[Matthew Turk]

Hi Augie,

On Tue, Feb 9, 2016 at 2:01 PM, Augie Fackler <raf at durin42.com> wrote:

> On Mon, Feb 08, 2016 at 03:55:59PM -0800, Sean Farley wrote:
> >
> > timeless <timeless at gmail.com> writes:
> >
> > > Sean Farley <sean at farley.io> wrote:
> > >> Yes, this is precisely the experience I have with C++ and libc++ (yuck
> > >> and gross incompatibility with libstdc++). It's not just Solaris,
> > >> either. Try anything that is not {Linux, Windows, Mac} / {gcc, clang}
> > >> and you'll see all kinds of corner cases crop up.
> > >
> > > There are plenty of painful corner cases on Windows too.
> > > And if you try hard enough you can get plenty of annoying corner cases
> on Linux.
> > >
> > > Would a CFront style approach be good enough? Write in limited C++,
> > > use llvm to convert it to C, commit the C++ as source and the
> > > generated C and have the build system use the generated C?
> >
> > I'd rather not go down this route. It's a little too close to switching
> > languages completely.
> >
> > By writing in C++ we are making it harder for us in the future to use
> > pypy. I think we can all agree that writing Python C is annoying
> > (opposed to just straight C). One way forward could be:
> >
> > - write standard C, creating a shared library
> > - use cffi for binding to that
> >
>
> Writing standard C is actually /worse/ than writing Python C. You have
> even fewer nice things to work with (no map type out of the box, but
> in Python C you can at least use PyDict). Part of the reason C++
> appeals is precisely because it's a widely-available language with
> better memory management tools (like smart pointers) and richer standard
> tools available (STL collections come to mind).
>
> I don't dispute the idea that it'll make some things harder. I *am*
> getting tired of the same old "HPC people can't get their shit
> together" song and dance being the reason I can't have any nice things
> and have to write a language that's fundamentally stuck in the 70s.
>
>
On IRC I mentioned this, but I wanted to bring it up here as well.  I work
on a python project that gets deployed on HPC clusters quite a lot, and we
have long been resistant to C++ in the codebase.  The reasons for this have
been similar to what Sean mentions (although I have not personally done
much with BG machines) and in particular are often related to things like
running on reduced OS nodes that don't have shared library loaders, etc etc.

This has in the past led to needing to do any number of hoop-jumping to
statically link a bunch of Python objects into a massive executable,
mangling zipimport, and so on.  Lately though, we have (in our project,
which may *not* translate elsewhere) found that these have not been issues;
for the most part, deployments are now on machines that don't restrict
nodes having shared libraries, things are slightly more up to date, etc
etc.  C++11 support can be a little weak in places, but I don't have the
numbers to back it up.  Statically linking can be really tricky with mixed
codebases that sometimes share different compiler toolchains (i.e., mixing
gcc with icc), which has caused us headaches previously.

But, as I mentioned to Sean on IRC, in our project we have opened up a
little bit to more C++ in the code base.  The machines we're finding are
now much more up to date on compiler toolchains and we're not really
needing to static link anymore.  For the use cases hg is used in, typically
individuals will be running on "head" nodes which are usually much more
feature rich anyway.

The biggest issues I have *personally* seen with mercurial on HPC centers
in the last year or two have been completely unrelated to mercurial.  These
come down to:

 * Old, out of date OpenSSL libraries or python installations
 * Bad filesystem performance for metadata lookups

I'll hold back from outright disagreeing with Sean (especially since IIRC
he's more experienced in the DOE HPC side of things, and I'm more
experienced in the NSF HPC side) but I don't think that from *my*
perspective C++ embedded in a Python project would be a big barrier to
adoption of hg on the HPC machines I use.  Although, come to think of it, I
use matplotlib on *every* machine I work on, and it has a large C++
component embedded in a Python project, and it has only been a problem when
running on compute nodes, which hg probably doesn't need to do.

-Matt


> Put another way: I'm totally open to having some way to do cleaner
> programming in C, but as it stands the tools we're able to use there
> are disastrously difficult and error-prone.
>
>
> >
> > The benefit is that we can use our C code for pypy. The downside is that
> > we introduce a cffi dependency.
> >
>
> http://doc.pypy.org/en/release-1.9/cppyy.html is a thing for pypy. I
> can't speak to its overall value or usabilty, but C++ does not
> preclude use of pypy in the future.
>
> > _______________________________________________
> > Mercurial-devel mailing list
> > Mercurial-devel at mercurial-scm.org
> > https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160209/5dfa7ea0/attachment.html>