MQ performance on large repo

[Adrian Buehlmann]

On 26.02.2010 23:30, Greg Ward wrote:
> On Fri, Feb 26, 2010 at 4:19 PM, Greg Ward <greg at gerg.ca> wrote:
>> I'm going to dig into the decodedir() calls, since that looks like the
>> lowest hanging fruit.  Suggestions are welcome.

Caveat. These fruits might not be that low-hanging, actually.

> Looks like fncache is the culprit.  Here's what my .hg/store/fncache looks like:
> 
>   $ wc -l .hg/store/fncache
>   247385 .hg/store/fncache
> 
> But:
> 
>   $ sort -u .hg/store/fncache | wc -l
>   28520
>   $ find .hg/store/data -type f | wc -l
>   26812
> 
> Lots of duplicate lines in that fncache!

Finding duplicate lines in fncache is by design.

> Looks like the performance hit boils down to store.fncachestore
> testing if a path is in fncache ("path not in fnc", store.py around
> line 304). That single __contains__() call accounts for a big chunk of
> the 7-8 sec runtime of qrefresh.  The last little bit of stack trace
> leading up to one of those 247385 calls to decodedir() is
> 
>   File "/home/gward/src/hg-crew/mercurial/repair.py", line 130, in strip
>     repo.sopener(file, 'a').truncate(troffset)
>   File "/home/gward/src/hg-crew/mercurial/store.py", line 304, in fncacheopener
>     and path not in fnc):
>   File "/home/gward/src/hg-crew/mercurial/store.py", line 283, in __contains__
>     self._load()
>   File "/home/gward/src/hg-crew/mercurial/store.py", line 266, in _load
>     self.entries.add(decodedir(line[:-1]))
> 
> ...so really, it's loading my 247385-line fncache that hurts.  And I
> expect it will hurt any call to repair.strip().
> 
> So, a possible workaround:
> 
>   $ rm .hg/store/fncache

This invalidates the repo, as detected by verify.

> Indeed, that speeds things up noticeably:
> 
>   * qrefresh: 5.0 .. 5.5 sec (from 7-8 sec)
>   * qpop: ~3.5 sec (from ~5 sec)
>   * qpush: ~1.5 sec (from ~4 sec)
> 
> Huh.  I've been reading the code and the fncacheRepoFormat wiki page,
> and remain unenlightened.  What purpose does .hg/store/fncache serve?

fncache repositories use a hash filename encoding function
(mercurial.store.hybridencode) which basically shortens longish filelog
"logical" filenames in the store inside .hg by encoding them using
sha-1. This function has no inverse, because sha-1 has no inverse
function. Revlog names with a length below some threshold are stored
unhashed.

This is done because some important crappy parts of Windows have a
ridiculous absolute maximum total path length of 260, even though WinNT
(and newer) basically is capable of taking path lengths up to 32,767
UTF-16 code words. There is even a special, separate, long-path API in
Win32, which almost no program uses (because of those other crappy parts
everyone on Windows has to live with).

There are some real world repos in the wild that actually -do- trigger
the filename hashing threshold (e.g. Netbeans, see example below).

fnache provides a mechanism for enumerating all filelog names of the
store without walking all manifest revisions.

Enumerating all filelog names of the store is used by uncompressed
cloning (store-file-copy-fast-cloning).

For non-fncache repos, uncompressed cloning enumerates the filelog names
in the store by walking the file system directory tree and applying an
inverse encoding function (mercurial.store.decodefilename) to the
filenames found. The filelog name encoding done by non-fncache repos
-does- have an inverse function.

IIRC uncompressed cloning sends a list of all (direncoded!) filelog
names in the store over the wire, followed by the revlog contents.

For fncache repos, that list of filelog names can't be compiled by
walking the filesystem directory tree, since the unencoded (non-hashed)
filelog name of a store file path of e.g. (a real example)

C:\Users\adi\hgrepos\reposamples\netbeans-main-golden\.hg\store\dh\j2ee.kit\test\qa-funct\data\freeform\cmp2\src\main\org\jboss\docs\crimeportcee7f51b82be7015f740d9fb32712156efe17ddf.i

can't be calculated from taking that path, because the sha-1 hash part
("7f51b82be...") has no inverse (no joking, they really do have a
"crimeport" :-).

The fncache repo format is the default since Mercurial 1.1, because
people *do* move repositories between windows and linux. Some projects
even access repos on the file level which reside on the same volume,
from both Windows and linux clients.