[PATCH 3 of 3 STABLE] py3: define and use json.loads polyfill

[Yuya Nishihara]

On Sat, 02 Nov 2019 12:19:58 -0700, Gregory Szorc wrote:
> # HG changeset patch
> # User Gregory Szorc <gregory.szorc at gmail.com>
> # Date 1572721775 25200
> #      Sat Nov 02 12:09:35 2019 -0700
> # Branch stable
> # Node ID b4a9220a4eb5a61891027d59cd0bded6dc5f7578
> # Parent  0d0cd63ca1702b901df2cc021d1f51c77bc0bf61
> py3: define and use json.loads polyfill

Queued for stable, thanks.

Just wondered if we can drop support for Python 3.5.

> +    # Python 3.5's json.load and json.loads require str. We polyfill its
> +    # code for detecting encoding from bytes.
> +    if sys.version_info[0:2] < (3, 6):
> +
> +        def _detect_encoding(b):
> +            bstartswith = b.startswith
> +            if bstartswith((codecs.BOM_UTF32_BE, codecs.BOM_UTF32_LE)):
> +                return 'utf-32'
> +            if bstartswith((codecs.BOM_UTF16_BE, codecs.BOM_UTF16_LE)):
> +                return 'utf-16'
> +            if bstartswith(codecs.BOM_UTF8):
> +                return 'utf-8-sig'
> +
> +            if len(b) >= 4:
> +                if not b[0]:
> +                    # 00 00 -- -- - utf-32-be
> +                    # 00 XX -- -- - utf-16-be
> +                    return 'utf-16-be' if b[1] else 'utf-32-be'
> +                if not b[1]:
> +                    # XX 00 00 00 - utf-32-le
> +                    # XX 00 00 XX - utf-16-le
> +                    # XX 00 XX -- - utf-16-le
> +                    return 'utf-16-le' if b[2] or b[3] else 'utf-32-le'
> +            elif len(b) == 2:
> +                if not b[0]:
> +                    # 00 XX - utf-16-be
> +                    return 'utf-16-be'
> +                if not b[1]:
> +                    # XX 00 - utf-16-le
> +                    return 'utf-16-le'
> +            # default
> +            return 'utf-8'

There should be no need to support encodings other than utf-8 (as Python 2.7
wouldn't), but it's copy-pasta so I don't care.