[PATCH] hgwebdir: return memory to the OS after each request

[Antoine Pitrou]

On Tue, 03 Jul 2012 11:39:28 +0200
Mads Kiilerich <mads at kiilerich.com> wrote:
> 
> I assume the problem is that there are some cyclic references that the 
> reference counting deallocator can't release so they are 'leaked' to the 
> garbage collector.

Probably, and by the time the garbage collector is called, some memory
has been allocated for another repository, causing memory fragmentation.

There are two layers in Python's object allocation:
- the small objects allocator returns blocks of memory from pools
  segregated by allocation size, themselves carved from bigger
  (256 KB) arenas
- the 256 KB arenas are requested from the OS using malloc() (and
  therefore relinquished using free()), which is a second opportunity
  for fragmentation

(note that in Python 3.3, arenas are requested using mmap() under Unix)

> How much difference do each of the 3 steps make?

I have not tried to investigate this very deeply. I know that
gc.collect() is necessary, otherwise the improvement is much smaller.
I'm not sure invalidate() is still useful if gc.collect() is called. It
probably doesn't hurt anyway (invalidate() doesn't look costly).

(doing experiments on a live server is a bit annoying)

> Wouldn't it be better to solve 
> it differently and as a more generic solution, for example by breaking 
> the cycles explicitly somewhere or using weak references so we can avoid 
> the workaround of starting the global garbage collector in hgweb?

It's common to have reference cycles in programs written in an
object-oriented style. Often objects will naturally refer to each
other. I agree using weakrefs would be theoretically more elegant,
however it may also break APIs, if used on e.g. public attributes.

Another solution than using weakrefs is a close() method that would
remove all potentially cyclic references.

Both these solutions, however, require much more knowledge of
Mercurial's internals than was necessary for the proposed patch :)
They are also more fragile since it's easy to add a new reference cycle
without noticing.

Regards

Antoine.