[PATCH 2 of 4] obsstore: disable garbage collection during initialisation (issue4456)

[Antoine Pitrou]

On Wed, 03 Dec 2014 20:00:23 -0600
Matt Mackall <mpm at selenic.com> wrote:
> On Thu, 2014-12-04 at 01:44 +0100, Antoine Pitrou wrote:
> > On Wed, 03 Dec 2014 17:31:25 -0600
> > Matt Mackall <mpm at selenic.com> wrote:
> > > On Sun, 2014-11-30 at 05:17 -0800, Pierre-Yves David wrote:
> > > > 
> > > > On 11/30/2014 05:06 AM, Antoine Pitrou wrote:
> > > > > On Sat, 29 Nov 2014 17:57:42 -0800
> > > > > Pierre-Yves David <pierre-yves.david at ens-lyon.org> wrote:
> > > > >> # HG changeset patch
> > > > >> # User Pierre-Yves David <pierre-yves.david at fb.com>
> > > > >> # Date 1417049911 28800
> > > > >> #      Wed Nov 26 16:58:31 2014 -0800
> > > > >> # Node ID d0f3dac4ea2b4aff51946c7db0834aa4e5c3e82a
> > > > >> # Parent  04eb7e49d2b6f90f71aa85de9ad0b4d70670d688
> > > > >> obsstore: disable garbage collection during initialisation (issue4456)
> > > > >>
> > > > >> Python garbage collection is triggered by contained creation. So code that
> > > > >> creates a lot of tuple tends to trigger GC a lot. We disable the gc during
> > > > >> obsolescence marker parsing and associated initialization. The provide and
> > > > >> interesting speedup (25%).
> > > > >>
> > > > >> On my 58758 markers repo:
> > > > >> before: 0.468247 seconds
> > > > >> after:  0.344362 seconds
> > > > >
> > > > > Which Python version is that?
> > > > 
> > > > Python 2.7.8 (default, Oct 18 2014, 12:50:18)
> > > > [GCC 4.9.1]
> > > 
> > > Python's GC behavior while building large containers is quite
> > > questionable (aka quadratic).
> > 
> > It shouldn't, see https://hg.python.org/cpython/rev/79276316b94b/
> > 
> > If you have a reproducer, please open an issue at bugs.python.org.
> 
> You're right, the quadratic behavior appears to be gone in 2.7. In 2.7,
> it's down to O(n): generic construction of lists of tuples gets about 4x
> slower. So it's still definitely worth disabling for any potentially
> large list or dict we might build. And obsolete markers are in that
> category.

Thanks. I'm wondering if there would be a way to improve our current
heuristics to alleviate such issues.

Regards

Antoine.