changing the module loader to handle python versions

[timeless]

So, I spent some time looking into this...

from the setup side:
get_ext_fullpath [hgbuildext] build_ext.py
  package_dir = os.path.abspath(build_py.get_package_dir(package))
  return os.path.join(package_dir, filename)

I don't see a good way to change this for only the .so / .dll files.
Plus, I don't see any particular reason to have to invalidate the
cache for the wrong version of python.

Having build_py.get_package_dir return a per python version seems
pretty easy. And then having the loader unconditionally (i.e. not just
for C code, but also python) favor the python versioned directory
should result in us being able to fairly easily maintain side-by-side
versions for various pythons.

FWIW, I'm well aware that I can make clean between things.
1. It's a PITA
2. It means that whenever I switch, my switch takes *much* longer.
3. Ideally I should be able to run these things independently without
them stepping on each other.

Thoughts?

On Tue, May 17, 2016 at 2:06 AM, Simon King <simon at simonking.org.uk> wrote:
> On Mon, May 16, 2016 at 5:29 PM, timeless <timeless at gmail.com> wrote:
>> Simon King wrote:
>>> I don't think that's supposed to happen, is it?
>>
>> Dunno. I don't claim to be a python expert.
>>
>> I do actively switch between at least py26 py27 py3 and to a lesser
>> extent pypy and some other flavors. So I hit all sorts of lousy edges.
>>
>>> Python should automatically invalidate .pyc files based on a magic number that
>>> changes when the format changes:
>>>
>>> https://hg.python.org/cpython/file/2.7/Python/import.c#l31
>>
>> (py26)[timeless at gcc2-power8 crew]$ make local
>> python setup.py  \
>>   build_py -c -d . \
>>   build_ext  -i \
>>   build_hgexe  -i \
>>   build_mo
>> running build_py
>> byte-compiling ./mercurial/byterange.py to byterange.pyc
>> byte-compiling ./mercurial/archival.py to archival.pyc
>> byte-compiling ./mercurial/crecord.py to crecord.pyc
>> byte-compiling ./mercurial/windows.py to windows.pyc
>> byte-compiling ./mercurial/lsprof.py to lsprof.pyc
>> byte-compiling ./mercurial/similar.py to similar.pyc
>> byte-compiling ./mercurial/scmwindows.py to scmwindows.pyc
>> byte-compiling ./mercurial/verify.py to verify.pyc
>> byte-compiling ./mercurial/py3kcompat.py to py3kcompat.pyc
>> byte-compiling ./mercurial/win32.py to win32.pyc
>> byte-compiling ./mercurial/__modulepolicy__.py to __modulepolicy__.pyc
>> byte-compiling ./mercurial/hgweb/webutil.py to webutil.pyc
>> byte-compiling ./mercurial/hgweb/wsgicgi.py to wsgicgi.pyc
>> byte-compiling ./mercurial/httpclient/socketutil.py to socketutil.pyc
>> byte-compiling ./hgext/convert/subversion.py to subversion.pyc
>> byte-compiling ./hgext/convert/convcmd.py to convcmd.pyc
>> byte-compiling ./hgext/convert/hg.py to hg.pyc
>> byte-compiling ./hgext/convert/cvsps.py to cvsps.pyc
>> byte-compiling ./hgext/convert/bzr.py to bzr.pyc
>> byte-compiling ./hgext/convert/filemap.py to filemap.pyc
>> byte-compiling ./hgext/convert/common.py to common.pyc
>> byte-compiling ./hgext/convert/gnuarch.py to gnuarch.pyc
>> byte-compiling ./hgext/convert/cvs.py to cvs.pyc
>> byte-compiling ./hgext/convert/monotone.py to monotone.pyc
>> byte-compiling ./hgext/fsmonitor/pywatchman/pybser.py to pybser.pyc
>> byte-compiling ./hgext/largefiles/wirestore.py to wirestore.pyc
>> byte-compiling ./hgext/largefiles/basestore.py to basestore.pyc
>> byte-compiling ./hgext/largefiles/lfutil.py to lfutil.pyc
>> byte-compiling ./hgext/largefiles/localstore.py to localstore.pyc
>> byte-compiling ./hgext/largefiles/proto.py to proto.pyc
>> byte-compiling ./hgext/largefiles/remotestore.py to remotestore.pyc
>> running build_ext
>> building 'mercurial.bdiff' extension
>> creating build/temp.linux-ppc64le-2.6
>> creating build/temp.linux-ppc64le-2.6/mercurial
>> gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall
>> -Wstrict-prototypes -fPIC
>> -I/home/timeless/.localpython26/include/python2.6 -c mercurial/bdiff.c
>> -o build/temp.linux-ppc64le-2.6/mercurial/bdiff.o
>> In file included from
>> /home/timeless/.localpython26/include/python2.6/Python.h:125:0,
>>                  from mercurial/bdiff.c:13:
>> /home/timeless/.localpython26/include/python2.6/modsupport.h:27:1:
>> warning: '_PyArg_ParseTuple_SizeT' is an unrecognized format function
>> type [-Wformat=]
>>  PyAPI_FUNC(int) PyArg_ParseTuple(PyObject *, const char *, ...)
>> Py_FORMAT_PARSETUPLE(PyArg_ParseTuple, 2, 3);
>>  ^
>> gcc -pthread -shared build/temp.linux-ppc64le-2.6/mercurial/bdiff.o -o
>> /home/timeless/hg/crew/mercurial/bdiff.so
>> running build_hgexe
>> running build_mo
>> env HGRCPATH= python hg version
>> Traceback (most recent call last):
>>   File "hg", line 41, in <module>
>>     mercurial.util.setbinary(fp)
>>   File "/home/timeless/hg/crew/mercurial/demandimport.py", line 130,
>> in __getattribute__
>>     self._load()
>>   File "/home/timeless/hg/crew/mercurial/demandimport.py", line 96, in _load
>>     mod = _hgextimport(_import, head, globals, locals, None, level)
>>   File "/home/timeless/hg/crew/mercurial/demandimport.py", line 53, in
>> _hgextimport
>>     return importfunc(name, globals, *args, **kwargs)
>>   File "/home/timeless/hg/crew/mercurial/util.py", line 2636, in <module>
>>     if safehasattr(parsers, 'dirs'):
>>   File "/home/timeless/hg/crew/mercurial/util.py", line 136, in safehasattr
>>     return getattr(thing, attr, _notset) is not _notset
>>   File "/home/timeless/hg/crew/mercurial/demandimport.py", line 130,
>> in __getattribute__
>>     self._load()
>>   File "/home/timeless/hg/crew/mercurial/demandimport.py", line 96, in _load
>>     mod = _hgextimport(_import, head, globals, locals, None, level)
>>   File "/home/timeless/hg/crew/mercurial/demandimport.py", line 53, in
>> _hgextimport
>>     return importfunc(name, globals, *args, **kwargs)
>>   File "/home/timeless/hg/crew/mercurial/__init__.py", line 143, in load_module
>>     mod = imp.load_module(name, *modinfo)
>> ImportError: Python minor version mismatch: The Mercurial extension
>> modules were compiled with Python 2.7.8, but Mercurial is currently
>> using Python with sys.hexversion=33950192: Python 2.6.9 (unknown, Apr
>> 13 2016, 12:40:12)
>> [GCC 4.9.2 20141101 (Red Hat 4.9.2-1)]
>>  at: /home/timeless/hg/py26/bin/python
>> Makefile:47: recipe for target 'local' failed
>> make: *** [local] Error 1
>>
>> I expect someone to hand waive this as a "build system issue", or as a
>> "compiled library problem". But from my perspective, switching between
>> python versions is not fun.
>
> Hmm, yes, I see what you mean. It's not a problem with .pyc files, and
> doesn't really have any relation to these patches, but it's clearly a
> problem. And indeed <handwaive>it does appear to be a build system
> issue</handwaive> - the mercurial/*.so files are
> python-version-dependent, but they don't get rebuilt when you run
> "make local" with a different python version.
>
> As a workaround, perhaps you could "rm mercurial/*.so; make local"?
>
> Simon