[PATCH 0 of 3] defer updating the fncache file to a single file open

[Adrian Buehlmann]

I've been watching files flying by the opener for quite some time now.
Recently, I even made a diagnostic extension that wraps the opener and
prints some debug info. I called it fsdebug and it's available at

  https://bitbucket.org/abuehl/fsdebug

in case anyone is interested.

It has bugged me for quite a while now how we open the fncache file for
each and every single new store entry. With fsdebug enabled this for
example goes like this:

    $ echo foo > a
    $ echo bla > b
    $ hg add
    adding a
    adding b
    opener: wb  -1 (atomicfile) 'dirstate'
    $ hg ci -mx
    opener: wb  -1 (atomicfile) 'branchheads.cache'
    opener: wb  -1   (unlinked) 'last-message.txt'
    opener: wb   0    (newfile) 'journal.dirstate'
    opener: wb   0    (newfile) 'journal.branch'
    opener: wb   0    (newfile) 'journal.desc'
    opener: ab   1     (normal) 'fncache'            <---- 1
    opener: a+b  0    (newfile) 'data/a.i'
    opener: ab   1     (normal) 'fncache'            <---- 2
    opener: a+b  0    (newfile) 'data/b.i'
    opener: a+b  1     (normal) '00manifest.i'
    opener: a+b  1     (normal) '00changelog.i'
    opener: ab   1     (normal) '00changelog.i'
    opener: wb  -1 (atomicfile) 'dirstate'

As you can see above, there are two opens of fncache, one per added file.

Today I thought we can do better and created this patch series. With the
series applied, the same use case goes like this:

    $ hg ci -mx
    opener: wb  -1   (unlinked) 'last-message.txt'
    opener: wb   0    (newfile) 'journal.dirstate'
    opener: wb   0    (newfile) 'journal.branch'
    opener: wb   0    (newfile) 'journal.desc'
    opener: a+b  0    (newfile) 'data/a.i'
    opener: a+b  0    (newfile) 'data/b.i'
    opener: a+b  1     (normal) '00manifest.i'
    opener: a+b  1     (normal) '00changelog.i'
    opener: ab   1     (normal) '00changelog.i'
    opener: wb  -1 (atomicfile) 'fncache'          <---- single open !
    opener: wb  -1 (atomicfile) 'dirstate'

As you can see, a single open for fncache near the end!

Of course this was only a toy use case. For large pulls there can be
quite a number of new files and thus an open on fncache for each new file.


The first patch inserts a couple sort calls in the tests. The order of
entries in the fncache file is not sorted.

The second patch wraps the contents of function localrepo.stream_in into
a lock acquisition try finally block. The patch looks more horrible than
it actually is, because it basically just indents a pile of lines by one
level. This was needed to prepare for patch 3.

Patch 3 is where the beef is. See the change message there.