[Bug 6075] New: explicit GC on every request make repositories with thousands of largesfiles very slow to clone

mercurial-bugs at mercurial-scm.org mercurial-bugs at mercurial-scm.org
Fri Feb 8 04:41:49 EST 2019


https://bz.mercurial-scm.org/show_bug.cgi?id=6075

            Bug ID: 6075
           Summary: explicit GC on every request make repositories with
                    thousands of largesfiles very slow to clone
           Product: Mercurial
           Version: 4.9
          Hardware: All
                OS: Linux
            Status: UNCONFIRMED
          Severity: bug
          Priority: wish
         Component: hgweb
          Assignee: bugzilla at mercurial-scm.org
          Reporter: lassea at unity3d.com
                CC: mercurial-devel at mercurial-scm.org

Created attachment 2032
  --> https://bz.mercurial-scm.org/attachment.cgi?id=2032&action=edit
demo output of lfpull from two hgwebs

in mercurial 4.5.3 this:

====
 finally:
    # There are known cycles in localrepository that prevent
    # those objects (and tons of held references) from being
    # collected through normal refcounting. We mitigate those
    # leaks by performing an explicit GC on every request.
    # TODO remove this once leaks are fixed.
    # TODO only run this on requests that create localrepository
    # instances instead of every request.
    gc.collect()
===

was added to "def run_wsgi(self, req, res):" in mercurial/hgweb/hgwebdir_mod.py
and is still present in latest version. This is a problem for repositories with
thousands of largesfile, as each largefile equals a request on initial clone.

at Unity We e.g. have a repository with ~17000 largefiles, and albeit that is a
pourly provisioned repostory, it takes ~15 minutes to clone from hwgeb running
on mercurial 4.5.2, and 2-4 hours on mercurial 4.5.3 and later which doenst
play well with our CI. 

I've made a docker image to debug this issue:

https://github.com/lasse-aagren/hgweb-gc-debug-docker.

If created as hgweb-demo:latest and run as:

docker run --rm -it hgweb-demo:latest /hgweb/lfpull-test.sh

it will spin up 2 hg serve instances, one on 4.5.2 and one on 4.5.3, on top a
demorepo with 1000 4k largefiles. It will then clone 3 consecutive time from
hgweb standalon on 4.5.2 and then 3 times from hgweb standalone on 4.5.3.

Result is attached, but the important points is, all 3 lfpulls from 4.5.2 takes
~60secs each. 

With 4.5.3 the first lfpull of the  1000 files takes ~60sec, the next one takes
~120 secs and the third ~180 secs. And it just get worse from there.

If I use mercurial 4.9 with the "gc.collect()" replaced by a "pass", it
performs as with version 4.5.2 and before.

Is my oberservations correct, and can something be done to mitigate it?

Please dont hesitate to ask me for further information.

Best regards,
Lasse Aagren

-- 
You are receiving this mail because:
You are on the CC list for the bug.


More information about the Mercurial-devel mailing list