D4053: [RFC]hgweb: garbage collect on every request in hgweb_mod too

[pulkit (Pulkit Goyal)]

pulkit created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  We recently updated to mercurial 4.6.2 on our servers and also increased number
  of requests to be made on server at one time. This made hgweb instance takes up
  a lot of memory, sometime upto 50GB of memory when requests are made parallely.
  
  This patch is motivated from
  https://www.mercurial-scm.org/repo/hg-committed/rev/ff2370a70fe8 which adds
  gc.collect() call for the server which serves multiple repositories.
  
  I tried profiling this fix, sometimes this shows less memory usage, sometimes
  more, so I am not sure whether this is right and hence RFC tag.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D4053

AFFECTED FILES
  mercurial/hgweb/hgweb_mod.py

CHANGE DETAILS

diff --git a/mercurial/hgweb/hgweb_mod.py b/mercurial/hgweb/hgweb_mod.py
--- a/mercurial/hgweb/hgweb_mod.py
+++ b/mercurial/hgweb/hgweb_mod.py
@@ -305,8 +305,19 @@
         with self._obtainrepo() as repo:
             profile = repo.ui.configbool('profiling', 'enabled')
             with profiling.profile(repo.ui, enabled=profile):
-                for r in self._runwsgi(req, res, repo):
-                    yield r
+                try:
+                    for r in self._runwsgi(req, res, repo):
+                        yield r
+                finally:
+                    # There are known cycles in localrepository that prevent
+                    # those objects (and tons of held references) from being
+                    # collected through normal refcounting. We mitigate those
+                    # leaks by performing an explicit GC on every request.
+                    # TODO remove this once leaks are fixed.
+                    # TODO only run this on requests that create localrepository
+                    # instances instead of every request.
+                    gc.collect()
+
 
     def _runwsgi(self, req, res, repo):
         rctx = requestcontext(self, repo, req, res)



To: pulkit, #hg-reviewers
Cc: mercurial-devel