I just started to get out my script for testing the commit performance, but I am stumbling over a strange memory-leak when using mercurial.dispatch.dispatch(req) or mercurial.commands.commit(…) After 7k commits, I see a memory-consumtion of about 3GiB, and rising. the part of the script which I think is relevant is here: https://bitbucket.org/ArneBab/hg-vs-git-for-server-apps/src/e57b2a1ca2a427e5f456ac5b6b5f5edc346f6235/run_test.py?at=default#cl-150 this is run in a long loop: https://bitbucket.org/ArneBab/hg-vs-git-for-server-apps/src/e57b2a1ca2a427e5f456ac5b6b5f5edc346f6235/run_test.py?at=default#cl-213 To test it, just clone the repo and execute make. https://bitbucket.org/ArneBab/hg-vs-git-for-server-apps (and watch the memory consumption rise to a few GiB within minutes)
Is this new? Can you bisect it?
Created attachment 1767 [details] leak.py
Created attachment 1768 [details] hgrc-reference
(In reply to comment #1) I did not bisect it yet, but I now created a minimal leaking example (see the attachments). This does not leak as heavily as the performance-scripts, but after 4k commits, it also reaches over 60 MiB for a 4MiB file (with lots of repetiton) and 5.7 MiB of history.
I was able to reproduce the leak with 3.0+303-9fb6f328576a. I hooked up guppy and was able to identify numerous leaks. The largest leaks are coming from str instances. I'm still tracking down exactly from where. We're also leaking filecacheentry and filecachesubentry instances.
I /think/ we may have a cycle at http://selenic.com/repo/hg/file/9fb6f328576a/mercurial/localrepo.py#l863. The filecacheentry instances all chain back to an "onclose" function that chains back to a transaction instance. Annotate says http://selenic.com/repo/hg/rev/cd443c7589cc is responsible for introducing onclose(). Sure enough, updating to p1(cd443c7589cc) (5dffd06f1e50) makes the leak go away.
Created attachment 1769 [details] objgraph for a leaked filecacheentry
Patch plugging the leak submitted to list.
Created attachment 1770 [details] leak.py A new version of leak.py which more nicely shows that Mercurial with the patch still leaks. To replicate: - Run the script. Note the memory usage. - Run the script again without deleting testrepo/. Notice that the memory usage starts lower than the first ended. On my system, the first run ends with a memory usage between 29MiB and 31MiB (oscillating). The second run after 500 commits is still between 23MiB and 24MiB. I hope that I used the patched mercurial. I only built it in the hg repo with make all and adjusted the paths in the script. I verified that req=request(["version"]); dispatch(req) gives 3.0+5-54d7657d7d1e+20140523).
Created attachment 1771 [details] leak_lockreleasing_memchecking.py I created a new version of the leak-script with two major changes: 1) It uses objgraph to check for memory usage. 2) It releases and reaquires the lock just before checking for memory usage. Disable the lock-release with --norelease. It instantly begins to leak one tuple, one function and one cell per commit. Also it creates the hgrc-reference, if that does not exist yet, so it is now really a standalone test.
In my performance test, I still see heavy leaking in both versions. The leakage is even worse when using dispatch. But when I release and reacquire the log, the objgraph does not show an increase of the number of objects, even though the required memory still rises sharply. 1000 revisions require 170MiB of memory with direct commit and relocking. With dispatch they require about 200MiB (there objgraph shows 1400 additional dicts in 100 revisions). Example for dispatch object growth in 100 revisions: dict 12420 +1400 list 25521 +1200 sortdict 4800 +600 config 2400 +300 set 1761 +200 request 800 +100 ui 800 +100 weakref 744 +2
Arne, when I apply Greg's patch and run your leak_lockreleasing_memchecking.py script, the leak disappears (4000 commits so far with no showgrowth output beyond commit 100). Are you sure you're running the script against your dev hg and not the system one?
I also started work on a new test to automatically scan for leaks. I'm still working out the details (things like the manifest LRU cache are introducing new objects that get reported as leaks). This will likely turn into a multiple patch series. In the mean time, feel free to experiment. http://hg.stage.mozaws.net/hg/rev/a903ec116060
Fixed by http://selenic.com/repo/hg/rev/99ba1d082287 Gregory Szorc <gregory.szorc@gmail.com> localrepo: prevent leak of transaction object (issue4258) The onclose() closure added in cd443c7589cc held a regular reference to the transaction object. This was causing the transaction to not gc and a leak to occur. The closure now holds a reference to the weakref instance and the leak goes away. (please test the fix)
(In reply to comment #12) Hi Durham, I also see no further show_growth after some time when I release and reaquire the lock, but still the memory consumption rises continually. I see the same, though to a much larger extend, in my performance test. Within just 2000 commits, the memory consumption rises to 300-400 MiB.
(In reply to comment #13) Hi Gregory, I still see the leak with hg 3.0.1. Can you tell me how I can access your experiments? Clicking the link only gets me to an empty page.
The link in comment #13 should be working again. The test doesn't work. But there is code there to run commits in a loop, which should reveal a leak. I won't have time for a bit to look into this.