Issue: import error on newest pypy on push (with hg-git resp. dulwich as extension)

[Matt Mackall]

On Wed, 2015-06-03 at 09:15 +0200, Sergey Brester wrote:
>  
> I use pypy, because for example by push from large hg-repo to large
> git-repo (hg-git + dulwich) using cpython, I can wait sometimes up to
> serveral minutes (dependent on commit size). With pypy the same is ready
> in few seconds.

That I can believe. I don't know how much optimization effort has gone
into dulwich. At Facebook, I believe we've swapped it out with libgit2.

> Additionaly, modern pypy jit wurm up very quickly and it byte code will
> be written (.pyc-file) by pypy also. 

Here's the thing:

$ time python -c "print 'hello world'"
hello world

real	0m0.013s <- sad
user	0m0.008s
sys	0m0.004s

$ time pypy -c "print 'hello world'"
hello world

real	0m0.033s <- awful
user	0m0.028s
sys	0m0.004s

$ time hg version -q
Mercurial Distributed SCM (version 3.4+473-4cc3fb23881d+20150531)

real	0m0.038s <- ugh
user	0m0.024s
sys	0m0.012s

I consider this latter number to be a huge embarrassment already. It
sets an absolute lower bound on any command we can do and dominates our
runtime for a lot of cases. We'd like to be 10x faster but Python isn't
even capable of that because it has a hard lower bound at .01s. So Pypy
is a big step in the wrong direction: it's just getting started when our
huge embarrassment benchmark is finishing.

And of course that embarrassing number is only possible because we've
invented our own demand-loading system to avoid a lot of module
searching and loading that Pypy would also be doing. With demandimport
disabled it looks like this:

$ time hg version -q
Mercurial Distributed SCM (version 3.4+473-4cc3fb23881d+20150531)

real	0m0.115s
user	0m0.096s
sys	0m0.016s

And here's where rock bottom is with Pypy with demandimport still
enabled:

$ time PYTHONPATH=/home/mpm/hg/mercurial/pure pypy hg version -q
Mercurial Distributed SCM (version 3.4+473-4cc3fb23881d+20150531)

real	0m0.100s
user	0m0.084s
sys	0m0.012s

Lastly, while we're on this topic:

$ time python3.4 -c "print('hello world')"
hello world

real	0m0.028s <- DO NOT WANT
user	0m0.024s
sys	0m0.000s

$ time perl -e "print 'hello world'"
hello world
real	0m0.002s <- the good old days
user	0m0.000s
sys	0m0.000s


> But pypy or not, such issue can occur on cpython also (for example using
> any hg-extension module). 

We're open to fixing issues here, but please understand that I would
much rather force developers to put the _nastiest, most awful hacks
imaginable_ in extensions than disable demandimport.

-- 
Mathematics is the supreme nostalgia of our time.