Dropping support for Python 2.4 and 2.5 / supporting Python 3

[Gregory Szorc]

I'd like to start a discussion that results in a clear plan for not
supporting old Python versions and also attempts to identify a plan to
support Mercurial running on Python 3.

Currently, Mercurial supports Python 2.4, 2.5, 2.6, and 2.7. There is
some lib2to3 code for converting source code to be Python 3 compliant.
But running Mercurial under Python 3 is still riddled with run-time
failures (mostly of the str/bytes/unicode sort) and true support is
likely a ways off (my toils over the weekend tell me it's really a
long ways off).

It is my understanding that the current desire to support older
Pythons stems from wanting to support RHEL and CentOS 5 (which ship
with Python 2.4). RHEL 5 has production support until April 2017 and
extended support until 2020. RHEL 6 ships with Python 2.6 and will
have production support until December 2020 and extended support until
2023. RHEL 7 is not yet released, but it will ship with Python 2.7.
Debian Squeeze also ships with Python 2.6. Standard support ends at
the end of this month. Extended support lasts until February 2016.

Meanwhile, CPython dropped all support for 2.4 in December 2008, 2.5
in May 2011, and 2.6 in October 2013. OS vendors (like RedHat)
continue to backport some fixes and issue updates. CPython 2.7 will be
officially supported until 2020 and likely longer by OS vendors.

Python 3.x is getting better with every minor version and there are
now some carrots that make it a very attractive platform. 3.5 will
offer yet more carrots, such as PEP-432 (which should improve startup
time) and PEP-460 (% and .format for bytes type). While most systems
in the wild will likely continue to have 2.7 and 3.x side-by-side, the
cries for 3.x support will only grow louder over time.

>From my experience with Python 3 porting, I can tell you that the
biggest problem is dealing with str/bytes/unicode types. I expect no
less with Mercurial. lib2to3 does a fantastic job of rewriting source
code to work with Python 3 for things like .iteritems(). But lib2to3
can't intelligently guess whether your types are str/bytes/unicode and
whether that type selection is the right choice. Your only realistic
recourse is to litter your code base with b'', u'', .encode(),
.decode() until everyone is happy (it's pretty easy to make Python 2
happy - less so with Python 3 since it doesn't perform automatic type
coercion and is more strict about not accepting bytes/str as arguments).

Unfortunately, Python 2.4 and 2.5 don't support b'' literals nor "from
__future__ import unicode_literals." So, you have to resort to foo
like six.b() and six.u() to declare binary and unicode types. Or, you
have to annotate only the string literals that shouldn't be converted
to b'' with lib2to3 via b() or some such. It's quite a huge mess. And
a lot of the ugliness and performance issues could be avoided if the
requirement for supporting 2.4 and 2.5 went away.

Having done some hacking around porting Mercurial to Python 3, I think
supporting 2.4/2.5 while porting to Python 3 is a fool's errand.

While continuing to support Python 2.4 for RHEL 5 is a noble goal, I
think it is one that should be abandoned ASAP. 2.4 and 2.5 (AFAICT
there are no strong reasons for supporting 2.5, just 2.4) are holding
back a lower effort port to Python 3. They are also preventing us from
using context managers (with statement), absolute imports,
try..except..finally, enhanced generators, modern exceptions (no
"except Exception, e" syntax), multiprocessing, str.format(), print()
function, abstract base classes, class decorators, namedtuple, and
numerous standard library enhancements and bug fixes/workarounds. Even
if you remove Python 3 from the equation, the language features alone
are compelling enough reasons to drop 2.4 and 2.5.

Is dropping support for 2.4, 2.5, and even 2.6 annoying for RHEL, etc
users? Yes. But I don't think it will be too bad in the wild. There
are tons of Python 2.7 RPMs and debs in existence. StackOverflow has
answers abound on how to configure them. Compiling and installing
Python in your home directory is also pretty straightforward. I reckon
Mercurial could distribute an archive with a statically-linked Python
bundled in (similar to how it's currently done on Windows). Even if a
RHEL, etc user didn't have access to a modern Python, they could still
run Mercurial 3.0 (or 3.1, 3.2, whatever the last release supporting
2.4/2.5 is). Given Mercurial's backwards compatible guarantees, they
would presumably still be able to communicate with Mercurial 4, 5, etc
servers for years ahead. Sure, they may not have access to bundle2,
changeset evolution, and other shiny new features and bug fixes, but
they will still be able to use Mercurial. If they really want those
compelling features, they can always install Python 2.7. And as more
and more projects drop support for <2.7, 2.7 will only become more
prevalent.

Python 2.4, 2.5, and 2.6 will drastically decrease in use in the
months and years ahead. 2.7 will linger for years and 3.x will
continue to see its share increase. I believe Mercurial should look
forward and drop support for at least 2.4 and 2.5 ASAP. If we're
courteous, maybe we wait for bundle2 and/or changeset evolution. But,
there will be bugs in those features and stranding people on buggy
initial versions is something to consider. Also, there will always be
something new and shiny in the pipe worth waiting for and thus always
an excuse to hold off. It's easy to fall into that trap and to be
stuck in the past.

As for supporting Python 3, I believe 2.4 and 2.5 are the primary
things holding us back. Next in line is the lack of % and .format() on
bytes. That should be fixed in Python 3.5 (assuming PEP-460 makes it).
Once we get that, I believe a port is realistically achievable. It's a
lot of work, but doable. Dropping 2.6 would make it a bit easier. But
I understand if there's reasons for wanting to retain 2.6 support for
another year or two (RHEL 6 being the biggie).

As an aside, I think there's an "activism" argument to be made as
well. While IMO Python has faltered with its version management and
left everyone with a world of hurt due to its rapid deprecation of old
versions and made the transition to 3.x harder than necessary (e.g.
not supporting u'' literals initially), it is what it is. Fortunately,
we now have the Python 2.7 support until 2020 guarantee. I think that
as a highly visible Python project, Mercurial should lead the way in
dropping Python <2.7. The world will be a better place with only one
Python 2.x version (2.7). If nothing else, the path towards 3.x is
much easier when only 2.7 is considered, so dropping <2.7 is a
considerable step towards increased 3.x adoption. As a popular Python
project, Mercurial is sitting in position to show leadership here.

Finally, I'd like us to consider the impact of supporting <2.7 on the
project. Newcomers to Python have and will continue to be learning
Python 2.7 or 3. I believe that us supporting 2.4 and even 2.6 will
turn potential contributors away. People prefer to hack on clean,
modern code bases as opposed to antiquated ones. Is there anyone among
us who wouldn't love to utilize some new Python 2.7 feature and reduce
the support surface area and the burden that accompanies it?

Python <2.7 is holding us back. Let's cut ourselves loose so we can
move faster.

Gregory