Line buffering not supported in binary mode on Python 3

[Gregory Szorc]

On Sat, Nov 16, 2019 at 4:26 PM Manuel Jacob <me at manueljacob.de> wrote:

> Python 3’s IO implementation doesn’t support line buffering in binary
> mode. On Python 3.8, this results in a RuntimeWarning, breaking
> test-ssh-proto.t.  Here are the tracebacks for the two occurences of the
> warning:
>
>    Traceback (most recent call last):
>      File "/home/manu/vcs/hg/hg", line 36, in <module>
>        dispatch.run()
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 111, in run
>        status = dispatch(req)
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 250, in
> dispatch
>        ret = _runcatch(req) or 0
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 424, in
> _runcatch
>        return _callcatch(ui, _runcatchfunc)
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 433, in
> _callcatch
>        return scmutil.callcatch(ui, func)
>      File "/home/manu/vcs/hg/mercurial/scmutil.py", line 177, in
> callcatch
>        return func()
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 414, in
> _runcatchfunc
>        return _dispatch(req)
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 1173, in
> _dispatch
>        return runcommand(
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 862, in
> runcommand
>        ret = _runcommand(ui, options, cmd, d)
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 1185, in
> _runcommand
>        return cmdfunc()
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 1171, in
> <lambda>
>        d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
>      File "/home/manu/vcs/hg/mercurial/util.py", line 1843, in check
>        return func(*args, **kwargs)
>      File "/home/manu/vcs/hg/mercurial/debugcommands.py", line 3205, in
> debugserve
>        logfh = os.fdopen(int(opts[b'logiofd']), r'ab', 1)
>      File "/usr/lib/python3.8/os.py", line 1021, in fdopen
>        return io.open(fd, *args, **kwargs)
>    RuntimeWarning: line buffering (buffering=1) isn't supported in binary
> mode, the default buffer size will be used
>
>    Traceback (most recent call last):
>      File "/home/manu/vcs/hg/hg", line 36, in <module>
>        dispatch.run()
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 111, in run
>        status = dispatch(req)
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 250, in
> dispatch
>        ret = _runcatch(req) or 0
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 424, in
> _runcatch
>        return _callcatch(ui, _runcatchfunc)
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 433, in
> _callcatch
>        return scmutil.callcatch(ui, func)
>      File "/home/manu/vcs/hg/mercurial/scmutil.py", line 177, in
> callcatch
>        return func()
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 414, in
> _runcatchfunc
>        return _dispatch(req)
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 1173, in
> _dispatch
>        return runcommand(
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 862, in
> runcommand
>        ret = _runcommand(ui, options, cmd, d)
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 1185, in
> _runcommand
>        return cmdfunc()
>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 1171, in
> <lambda>
>        d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
>      File "/home/manu/vcs/hg/mercurial/util.py", line 1843, in check
>        return func(*args, **kwargs)
>      File "/home/manu/vcs/hg/mercurial/debugcommands.py", line 3212, in
> debugserve
>        logfh = open(opts[b'logiofile'], b'ab', 1)
>      File "/home/manu/vcs/hg/mercurial/pycompat.py", line 296, in open
>        return builtins.open(name, sysstr(mode), buffering, encoding)
>    RuntimeWarning: line buffering (buffering=1) isn't supported in binary
> mode, the default buffer size will be used
>
> Possible ways to solve this:
>
> 1) Ensure that the users of the file object (seemingly only the logging
> file object) flush at all the necessary places themselves, and configure
> the file object to use the default buffering behavior.
> 2) Write our own buffering wrapper class that supports line buffering in
> binary mode.
> 3) Change the logging file object to use unicode (at least on Python 3).
>

Thank you for the bug report. This is indeed a valid incompatibility with
Python 3.8!

While Mercurial's test harness passes cleanly on 3.6 and 3.7, there are a
small number of test failures remaining on 3.8 (and 3.5). You can see all
these failures https://ci.hg.gregoryszorc.com/ (a very alpha CI system
which I maintain). Fortunately, the failure reported in your bug report was
captured and we were already aware of it! If you use Mercurial with Python
3.8, the only user-visible issue we're aware of is Python complaining about
unhandled exceptions at process termination (as seen in
https://ci.hg.gregoryszorc.com/job-info/hg-committed-da5ccc591cfff7d2fe421b71be8298ba9e492b25-debian10-cpython-3.8-0#failure-test-worker.t).
I suspect most users won't notice and that Mercurial is effectively stable
on 3.8. But if you want to play it "by the book," stick with 3.6 or 3.7 for
now, as those are the only Python versions that have clean test harness
runs.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20191118/1065e40c/attachment.html>