MQ performance on large repo

[Greg Ward]

On Sat, Feb 27, 2010 at 6:09 AM, Dirkjan Ochtman <dirkjan at ochtman.nl> wrote:
> On Fri, Feb 26, 2010 at 16:19, Greg Ward <greg at gerg.ca> wrote:
>> I'm using MQ on a large repo (106k changesets, 18k files) and it's a bit slow:
>>
>>  * qrefresh: 7-8 sec
>>  * qpop: ~5 sec
>>  * qpush: ~4 sec
>>
>> All three operations are dominated by user-space CPU time.
>>
>> So I thought I'd profile it a bit, starting with qrefresh since it's
>> the slowest.  Some interesting numbers jump out; the most expensive
>
> You might also want to check for a regression, by trying 1.2, 1.3, 1.4
> and current crew on your repository.

Not sure if that would help.  Since the biggest performance hit is
parsing .hg/store/fncache, and that file contains 10x as many lines as
it needs to, perhaps the culprit is how duplicates build up in fncache
over time.  I.e. you could argue that the performance bug is in the
code that I've been using to maintain
(qpop/qpush/qrefresh/pull/update/commit/push) this repo over the past
several weeks.  Trying 1.2/1.3/1.4 versions of the code for parsing an
unnecessarily large fncache seems like a distraction.

And, even if we optimize the be-jeezus out of parsing fncache, it
seems to me more sensible to avoid the extra work in the first place
by not writing so many duplicate lines in fncache.

Greg