Line buffering not supported in binary mode on Python 3

Manuel Jacob me at manueljacob.de
Tue Nov 19 03:45:44 EST 2019


On 2019-11-19 05:22, Gregory Szorc wrote:
> On Sat, Nov 16, 2019 at 4:26 PM Manuel Jacob <me at manueljacob.de> wrote:
> 
>> Python 3’s IO implementation doesn’t support line buffering in binary
>> mode. On Python 3.8, this results in a RuntimeWarning, breaking
>> test-ssh-proto.t.  Here are the tracebacks for the two occurences of 
>> the
>> warning:
>> 
>>    Traceback (most recent call last):
>>      File "/home/manu/vcs/hg/hg", line 36, in <module>
>>        dispatch.run()
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 111, in run
>>        status = dispatch(req)
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 250, in
>> dispatch
>>        ret = _runcatch(req) or 0
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 424, in
>> _runcatch
>>        return _callcatch(ui, _runcatchfunc)
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 433, in
>> _callcatch
>>        return scmutil.callcatch(ui, func)
>>      File "/home/manu/vcs/hg/mercurial/scmutil.py", line 177, in
>> callcatch
>>        return func()
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 414, in
>> _runcatchfunc
>>        return _dispatch(req)
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 1173, in
>> _dispatch
>>        return runcommand(
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 862, in
>> runcommand
>>        ret = _runcommand(ui, options, cmd, d)
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 1185, in
>> _runcommand
>>        return cmdfunc()
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 1171, in
>> <lambda>
>>        d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
>>      File "/home/manu/vcs/hg/mercurial/util.py", line 1843, in check
>>        return func(*args, **kwargs)
>>      File "/home/manu/vcs/hg/mercurial/debugcommands.py", line 3205, 
>> in
>> debugserve
>>        logfh = os.fdopen(int(opts[b'logiofd']), r'ab', 1)
>>      File "/usr/lib/python3.8/os.py", line 1021, in fdopen
>>        return io.open(fd, *args, **kwargs)
>>    RuntimeWarning: line buffering (buffering=1) isn't supported in 
>> binary
>> mode, the default buffer size will be used
>> 
>>    Traceback (most recent call last):
>>      File "/home/manu/vcs/hg/hg", line 36, in <module>
>>        dispatch.run()
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 111, in run
>>        status = dispatch(req)
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 250, in
>> dispatch
>>        ret = _runcatch(req) or 0
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 424, in
>> _runcatch
>>        return _callcatch(ui, _runcatchfunc)
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 433, in
>> _callcatch
>>        return scmutil.callcatch(ui, func)
>>      File "/home/manu/vcs/hg/mercurial/scmutil.py", line 177, in
>> callcatch
>>        return func()
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 414, in
>> _runcatchfunc
>>        return _dispatch(req)
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 1173, in
>> _dispatch
>>        return runcommand(
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 862, in
>> runcommand
>>        ret = _runcommand(ui, options, cmd, d)
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 1185, in
>> _runcommand
>>        return cmdfunc()
>>      File "/home/manu/vcs/hg/mercurial/dispatch.py", line 1171, in
>> <lambda>
>>        d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
>>      File "/home/manu/vcs/hg/mercurial/util.py", line 1843, in check
>>        return func(*args, **kwargs)
>>      File "/home/manu/vcs/hg/mercurial/debugcommands.py", line 3212, 
>> in
>> debugserve
>>        logfh = open(opts[b'logiofile'], b'ab', 1)
>>      File "/home/manu/vcs/hg/mercurial/pycompat.py", line 296, in open
>>        return builtins.open(name, sysstr(mode), buffering, encoding)
>>    RuntimeWarning: line buffering (buffering=1) isn't supported in 
>> binary
>> mode, the default buffer size will be used
>> 
>> Possible ways to solve this:
>> 
>> 1) Ensure that the users of the file object (seemingly only the 
>> logging
>> file object) flush at all the necessary places themselves, and 
>> configure
>> the file object to use the default buffering behavior.
>> 2) Write our own buffering wrapper class that supports line buffering 
>> in
>> binary mode.
>> 3) Change the logging file object to use unicode (at least on Python 
>> 3).
>> 
> 
> Thank you for the bug report. This is indeed a valid incompatibility 
> with
> Python 3.8!
> 
> While Mercurial's test harness passes cleanly on 3.6 and 3.7, there are 
> a
> small number of test failures remaining on 3.8 (and 3.5). You can see 
> all
> these failures https://ci.hg.gregoryszorc.com/ (a very alpha CI system
> which I maintain). Fortunately, the failure reported in your bug report 
> was
> captured and we were already aware of it! If you use Mercurial with 
> Python
> 3.8, the only user-visible issue we're aware of is Python complaining 
> about
> unhandled exceptions at process termination (as seen in
> https://ci.hg.gregoryszorc.com/job-info/hg-committed-da5ccc591cfff7d2fe421b71be8298ba9e492b25-debian10-cpython-3.8-0#failure-test-worker.t).
> I suspect most users won't notice and that Mercurial is effectively 
> stable
> on 3.8. But if you want to play it "by the book," stick with 3.6 or 3.7 
> for
> now, as those are the only Python versions that have clean test harness
> runs.

Hi Gregory,

I noticed this bug while looking at the CI system you mentioned 
specifically for bugs that I could fix.  Arch Linux uses Python 3.8 now 
and while its Mercurial version still runs on Python 2.7, I’d like to 
help to clear out some impediments that stop it from running on top of 
Python 3.8.

I’m leaning towards implementing variant (1) mentioned in my original 
email, unless there are objections, and send a patch in the next days.

-Manuel


More information about the Mercurial-devel mailing list