[PATCH 04 of 11] py3: byteify the LFS blobstore module

[Matt Harbison]

On Mon, 28 Jan 2019 06:41:01 -0500, Yuya Nishihara <yuya at tcha.org> wrote:

> On Mon, 28 Jan 2019 00:20:50 -0500, Matt Harbison wrote:
>> # HG changeset patch
>> # User Matt Harbison <matt_harbison at yahoo.com>
>> # Date 1548620368 18000
>> #      Sun Jan 27 15:19:28 2019 -0500
>> # Node ID d639ccd9f7139991165d04b4601b06c3341a79cd
>> # Parent  9a35d8754bdfaac20d1931dbd6a82233d11f4722
>> py3: byteify the LFS blobstore module
>
>> @@ -303,29 +305,32 @@ class _gitlfsremote(object):
>>                  rawjson = rsp.read()
>>          except util.urlerr.httperror as ex:
>>              hints = {
>> -                400: _('check that lfs serving is enabled on %s and  
>> "%s" is '
>> +                400: _(b'check that lfs serving is enabled on %s and  
>> "%s" is '
>>                         'supported') % (self.baseurl, action),
>> -                404: _('the "lfs.url" config may be used to override  
>> %s')
>> +                404: _(b'the "lfs.url" config may be used to override  
>> %s')
>>                         % self.baseurl,
>>              }
>> -            hint = hints.get(ex.code, _('api=%s, action=%s') % (url,  
>> action))
>> -            raise LfsRemoteError(_('LFS HTTP error: %s') % ex,  
>> hint=hint)
>> +            hint = hints.get(ex.code, _(b'api=%s, action=%s') % (url,  
>> action))
>> +            raise LfsRemoteError(
>> +                _(b'LFS HTTP error: %s') % stringutil.forcebytestr(ex),
>> +                  hint=hint)
>>          except util.urlerr.urlerror as ex:
>> -            hint = (_('the "lfs.url" config may be used to override  
>> %s')
>> +            hint = (_(b'the "lfs.url" config may be used to override  
>> %s')
>>                      % self.baseurl)
>> -            raise LfsRemoteError(_('LFS error: %s') %  
>> _urlerrorreason(ex),
>> +            raise LfsRemoteError(_(b'LFS error: %s') %  
>> _urlerrorreason(ex),
>>                                   hint=hint)
>>          try:
>>              response = json.loads(rawjson)
>>          except ValueError:
>> -            raise LfsRemoteError(_('LFS server returns invalid JSON:  
>> %s')
>> -                                 % rawjson)
>> +            raise LfsRemoteError(_(b'LFS server returns invalid JSON:  
>> %s')
>> +                                 % rawjson.encode("utf-8"))
>
> Does rsp.read() return a unicode string? If it doesn't, this could crash  
> if
> rawjson wasn't ASCII bytes.
>
>>
>>          if self.ui.debugflag:
>> -            self.ui.debug('Status: %d\n' % rsp.status)
>> +            self.ui.debug(b'Status: %d\n' % rsp.status)
>>              # lfs-test-server and hg serve return headers in different  
>> order
>> -            self.ui.debug('%s\n'
>> -                          %  
>> '\n'.join(sorted(str(rsp.info()).splitlines())))
>> +            headers = pycompat.bytestr(rsp.info())
>> +            self.ui.debug(b'%s\n'
>> +                          % b'\n'.join(sorted(headers.splitlines())))
>
> I'm not sure if rsp.info() never contains non-ASCII bytes.

I'm not sure about either of these.  I managed to hit one of these cases  
at work late Friday with py2.  I tried testing with py3, but it looks like  
authentication stuff is messed up with bytes vs str in the username:

   10.30.1.22 - - [01/Feb/2019:15:58:18 -0500]
   "GET /hg/installanywhere/configtool_install?cmd=capabilities HTTP/1.1"  
401
   401
   10.30.1.22 - b'mharbison' [01/Feb/2019:15:58:18 -0500]
   "GET /hg/installanywhere/configtool_install?cmd=capabilities HTTP/1.1"  
401
   401

I see where `%s - - [%s]` comes from, but not `- %s` or `- %r`, so I think  
this is from Apache itself.  retry_http_basic_auth() in url.py looks OK,  
though the password manager seems to be a mess of bytes vs str.  I can't  
seem to reproduce this with existing tests.

At any rate, I need to sort this out (and why it is failing on py2 in the  
first place) before I can see what's going on here.