[PATCH 1 of 3 RFC] mercurial: implement a source transforming module loader on Python 3

[Simon King]

On Mon, May 16, 2016 at 5:29 PM, timeless <timeless at gmail.com> wrote:
> Simon King wrote:
>> I don't think that's supposed to happen, is it?
>
> Dunno. I don't claim to be a python expert.
>
> I do actively switch between at least py26 py27 py3 and to a lesser
> extent pypy and some other flavors. So I hit all sorts of lousy edges.
>
>> Python should automatically invalidate .pyc files based on a magic number that
>> changes when the format changes:
>>
>> https://hg.python.org/cpython/file/2.7/Python/import.c#l31
>
> (py26)[timeless at gcc2-power8 crew]$ make local
> python setup.py  \
>   build_py -c -d . \
>   build_ext  -i \
>   build_hgexe  -i \
>   build_mo
> running build_py
> byte-compiling ./mercurial/byterange.py to byterange.pyc
> byte-compiling ./mercurial/archival.py to archival.pyc
> byte-compiling ./mercurial/crecord.py to crecord.pyc
> byte-compiling ./mercurial/windows.py to windows.pyc
> byte-compiling ./mercurial/lsprof.py to lsprof.pyc
> byte-compiling ./mercurial/similar.py to similar.pyc
> byte-compiling ./mercurial/scmwindows.py to scmwindows.pyc
> byte-compiling ./mercurial/verify.py to verify.pyc
> byte-compiling ./mercurial/py3kcompat.py to py3kcompat.pyc
> byte-compiling ./mercurial/win32.py to win32.pyc
> byte-compiling ./mercurial/__modulepolicy__.py to __modulepolicy__.pyc
> byte-compiling ./mercurial/hgweb/webutil.py to webutil.pyc
> byte-compiling ./mercurial/hgweb/wsgicgi.py to wsgicgi.pyc
> byte-compiling ./mercurial/httpclient/socketutil.py to socketutil.pyc
> byte-compiling ./hgext/convert/subversion.py to subversion.pyc
> byte-compiling ./hgext/convert/convcmd.py to convcmd.pyc
> byte-compiling ./hgext/convert/hg.py to hg.pyc
> byte-compiling ./hgext/convert/cvsps.py to cvsps.pyc
> byte-compiling ./hgext/convert/bzr.py to bzr.pyc
> byte-compiling ./hgext/convert/filemap.py to filemap.pyc
> byte-compiling ./hgext/convert/common.py to common.pyc
> byte-compiling ./hgext/convert/gnuarch.py to gnuarch.pyc
> byte-compiling ./hgext/convert/cvs.py to cvs.pyc
> byte-compiling ./hgext/convert/monotone.py to monotone.pyc
> byte-compiling ./hgext/fsmonitor/pywatchman/pybser.py to pybser.pyc
> byte-compiling ./hgext/largefiles/wirestore.py to wirestore.pyc
> byte-compiling ./hgext/largefiles/basestore.py to basestore.pyc
> byte-compiling ./hgext/largefiles/lfutil.py to lfutil.pyc
> byte-compiling ./hgext/largefiles/localstore.py to localstore.pyc
> byte-compiling ./hgext/largefiles/proto.py to proto.pyc
> byte-compiling ./hgext/largefiles/remotestore.py to remotestore.pyc
> running build_ext
> building 'mercurial.bdiff' extension
> creating build/temp.linux-ppc64le-2.6
> creating build/temp.linux-ppc64le-2.6/mercurial
> gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall
> -Wstrict-prototypes -fPIC
> -I/home/timeless/.localpython26/include/python2.6 -c mercurial/bdiff.c
> -o build/temp.linux-ppc64le-2.6/mercurial/bdiff.o
> In file included from
> /home/timeless/.localpython26/include/python2.6/Python.h:125:0,
>                  from mercurial/bdiff.c:13:
> /home/timeless/.localpython26/include/python2.6/modsupport.h:27:1:
> warning: '_PyArg_ParseTuple_SizeT' is an unrecognized format function
> type [-Wformat=]
>  PyAPI_FUNC(int) PyArg_ParseTuple(PyObject *, const char *, ...)
> Py_FORMAT_PARSETUPLE(PyArg_ParseTuple, 2, 3);
>  ^
> gcc -pthread -shared build/temp.linux-ppc64le-2.6/mercurial/bdiff.o -o
> /home/timeless/hg/crew/mercurial/bdiff.so
> running build_hgexe
> running build_mo
> env HGRCPATH= python hg version
> Traceback (most recent call last):
>   File "hg", line 41, in <module>
>     mercurial.util.setbinary(fp)
>   File "/home/timeless/hg/crew/mercurial/demandimport.py", line 130,
> in __getattribute__
>     self._load()
>   File "/home/timeless/hg/crew/mercurial/demandimport.py", line 96, in _load
>     mod = _hgextimport(_import, head, globals, locals, None, level)
>   File "/home/timeless/hg/crew/mercurial/demandimport.py", line 53, in
> _hgextimport
>     return importfunc(name, globals, *args, **kwargs)
>   File "/home/timeless/hg/crew/mercurial/util.py", line 2636, in <module>
>     if safehasattr(parsers, 'dirs'):
>   File "/home/timeless/hg/crew/mercurial/util.py", line 136, in safehasattr
>     return getattr(thing, attr, _notset) is not _notset
>   File "/home/timeless/hg/crew/mercurial/demandimport.py", line 130,
> in __getattribute__
>     self._load()
>   File "/home/timeless/hg/crew/mercurial/demandimport.py", line 96, in _load
>     mod = _hgextimport(_import, head, globals, locals, None, level)
>   File "/home/timeless/hg/crew/mercurial/demandimport.py", line 53, in
> _hgextimport
>     return importfunc(name, globals, *args, **kwargs)
>   File "/home/timeless/hg/crew/mercurial/__init__.py", line 143, in load_module
>     mod = imp.load_module(name, *modinfo)
> ImportError: Python minor version mismatch: The Mercurial extension
> modules were compiled with Python 2.7.8, but Mercurial is currently
> using Python with sys.hexversion=33950192: Python 2.6.9 (unknown, Apr
> 13 2016, 12:40:12)
> [GCC 4.9.2 20141101 (Red Hat 4.9.2-1)]
>  at: /home/timeless/hg/py26/bin/python
> Makefile:47: recipe for target 'local' failed
> make: *** [local] Error 1
>
> I expect someone to hand waive this as a "build system issue", or as a
> "compiled library problem". But from my perspective, switching between
> python versions is not fun.

Hmm, yes, I see what you mean. It's not a problem with .pyc files, and
doesn't really have any relation to these patches, but it's clearly a
problem. And indeed <handwaive>it does appear to be a build system
issue</handwaive> - the mercurial/*.so files are
python-version-dependent, but they don't get rebuilt when you run
"make local" with a different python version.

As a workaround, perhaps you could "rm mercurial/*.so; make local"?

Simon