MQ performance on large repo

[Steve Borho]

On Fri, Feb 26, 2010 at 4:30 PM, Greg Ward <greg at gerg.ca> wrote:
> On Fri, Feb 26, 2010 at 4:19 PM, Greg Ward <greg at gerg.ca> wrote:
>> I'm going to dig into the decodedir() calls, since that looks like the
>> lowest hanging fruit.  Suggestions are welcome.
>
> Looks like fncache is the culprit.  Here's what my .hg/store/fncache looks like:
>
>  $ wc -l .hg/store/fncache
>  247385 .hg/store/fncache
>
> But:
>
>  $ sort -u .hg/store/fncache | wc -l
>  28520
>  $ find .hg/store/data -type f | wc -l
>  26812
>
> Lots of duplicate lines in that fncache!
>
> Looks like the performance hit boils down to store.fncachestore
> testing if a path is in fncache ("path not in fnc", store.py around
> line 304). That single __contains__() call accounts for a big chunk of
> the 7-8 sec runtime of qrefresh.  The last little bit of stack trace
> leading up to one of those 247385 calls to decodedir() is
>
>  File "/home/gward/src/hg-crew/mercurial/repair.py", line 130, in strip
>    repo.sopener(file, 'a').truncate(troffset)
>  File "/home/gward/src/hg-crew/mercurial/store.py", line 304, in fncacheopener
>    and path not in fnc):
>  File "/home/gward/src/hg-crew/mercurial/store.py", line 283, in __contains__
>    self._load()
>  File "/home/gward/src/hg-crew/mercurial/store.py", line 266, in _load
>    self.entries.add(decodedir(line[:-1]))
>
> ...so really, it's loading my 247385-line fncache that hurts.  And I
> expect it will hurt any call to repair.strip().
>
> So, a possible workaround:
>
>  $ rm .hg/store/fncache
>
> Indeed, that speeds things up noticeably:
>
>  * qrefresh: 5.0 .. 5.5 sec (from 7-8 sec)
>  * qpop: ~3.5 sec (from ~5 sec)
>  * qpush: ~1.5 sec (from ~4 sec)
>
> Huh.  I've been reading the code and the fncacheRepoFormat wiki page,
> and remain unenlightened.  What purpose does .hg/store/fncache serve?

See format.usefncache in hgrc.5

--
Steve Borho