[PATCH] py3: handling .iteritems() vs .items()

Gregory Szorc gregory.szorc at gmail.com
Mon Jun 6 21:18:31 EDT 2016


On Sun, Jun 5, 2016 at 8:32 AM, Pulkit Goyal <7895pulkit at gmail.com> wrote:

> # HG changeset patch
> # User Pulkit Goyal <7895pulkit at gmail.com>
> # Date 1465140428 -19800
> #      Sun Jun 05 20:57:08 2016 +0530
> # Node ID ae89177a49c51f66962202598a44643c1dd1d18f
> # Parent  118a605e3ad9e1d30c4fd8bacc8310167ae1f222
> py3: handling .iteritems() vs .items()
>
> Using dict.items() instead of dict.iteritems() in
> py2 just for py3 compatibilty is not a good idea because
> in py2 dict.items() returns a copy of dictionary's list of
> pairs. This will take a lot of memory when dictionary is
> large.
>
> So it will be good to use .viewitems() in py2, so that we can iterate
> more than once and .items() in py3 as rest are gone. The .items() in
> py3 has an improved implementation.
>
> Importing from the util mdoule adds up the call cost which can be
> mitigated by using methodcaller. This approach can be used for
> mercurial/* and hgext/* modules.
> In contrib we have to import mercurial/util.py, that will be costly
> so handling it separately will be better. Also using .items() will
> be costly in contrib. So we have to add this hack everywhere or find
> another way out.
>
> I will like to get more advice on this as to what can be improvised
> in this approach and how can we handle the contrib section separately.
>

If it gets accepted, I would propose extending my custom module loader on
Python 3 to rewrite source code to accommodate API differences like
items/iteritems and range/xrange. See e.g.
https://hg.mozilla.org/users/gszorc_mozilla.com/hg/rev/7b72cfda136e

This would mean keeping the source Python 2 compatible and rewriting
"iteritems()" to "items()" on Python 3. This will require implementing
items() on our custom classes. I think that is a lesser evil than trying to
figure out when we should use one form or the other in our source.


>
> diff --git a/mercurial/util.py b/mercurial/util.py
> --- a/mercurial/util.py
> +++ b/mercurial/util.py
> @@ -23,6 +23,7 @@
>  import gc
>  import hashlib
>  import imp
> +import operator
>  import os
>  import re as remod
>  import shutil
> @@ -58,6 +59,8 @@
>  # This line is to make pyflakes happy:
>  urlreq = pycompat.urlreq
>
> +methodcaller = operator.methodcaller
> +
>  if os.name == 'nt':
>      from . import windows as platform
>  else:
> @@ -2830,3 +2833,12 @@
>
>  # convenient shortcut
>  dst = debugstacktrace
> +
> +def viewitems(dict):
> +
> +    # using methodcaller avoids having to create another Python call
> frame.
> +    if safehasattr(dict, 'viewitems'):
> +        viewitems = methodcaller('viewitems')
> +    else:
> +        viewitems = methodcaller('items')
> +    return viewitems(dict)
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160606/adce4b3c/attachment.html>


More information about the Mercurial-devel mailing list