MQ performance on large repo

[Greg Ward]

On Fri, Feb 26, 2010 at 4:19 PM, Greg Ward <greg at gerg.ca> wrote:
> I'm going to dig into the decodedir() calls, since that looks like the
> lowest hanging fruit.  Suggestions are welcome.

Looks like fncache is the culprit.  Here's what my .hg/store/fncache looks like:

  $ wc -l .hg/store/fncache
  247385 .hg/store/fncache

But:

  $ sort -u .hg/store/fncache | wc -l
  28520
  $ find .hg/store/data -type f | wc -l
  26812

Lots of duplicate lines in that fncache!

Looks like the performance hit boils down to store.fncachestore
testing if a path is in fncache ("path not in fnc", store.py around
line 304). That single __contains__() call accounts for a big chunk of
the 7-8 sec runtime of qrefresh.  The last little bit of stack trace
leading up to one of those 247385 calls to decodedir() is

  File "/home/gward/src/hg-crew/mercurial/repair.py", line 130, in strip
    repo.sopener(file, 'a').truncate(troffset)
  File "/home/gward/src/hg-crew/mercurial/store.py", line 304, in fncacheopener
    and path not in fnc):
  File "/home/gward/src/hg-crew/mercurial/store.py", line 283, in __contains__
    self._load()
  File "/home/gward/src/hg-crew/mercurial/store.py", line 266, in _load
    self.entries.add(decodedir(line[:-1]))

...so really, it's loading my 247385-line fncache that hurts.  And I
expect it will hurt any call to repair.strip().

So, a possible workaround:

  $ rm .hg/store/fncache

Indeed, that speeds things up noticeably:

  * qrefresh: 5.0 .. 5.5 sec (from 7-8 sec)
  * qpop: ~3.5 sec (from ~5 sec)
  * qpush: ~1.5 sec (from ~4 sec)

Huh.  I've been reading the code and the fncacheRepoFormat wiki page,
and remain unenlightened.  What purpose does .hg/store/fncache serve?

Greg