D5296: store: don't read the whole fncache in memory
pulkit (Pulkit Goyal)
phabricator at mercurial-scm.org
Wed Feb 27 10:43:40 EST 2019
pulkit added a comment.
In https://phab.mercurial-scm.org/D5296#87892, @yuja wrote:
> (resend without the "On ... wrote:" line)
> > Seeing the performance benefit it brings on our repo, I want to try other ways we can do this. Do we like having a conditional which checks the size of fncache and choose one of the approaches depending on how large it is?
> > If size(fncache) < 50M, use the current approach
> > else, use the approach described in this patch
> Suppose the current code is fast because it avoids running Python in loop,
> I think it can be extended to not use too much memory.
> chunk = b''
> while True:
> chunk += fp.read(chunk_size) # maybe ~10MB?
> p = chunk.rindex(b'\n') # need to handle error
> decodedir(chunk[:p + 1])...
> chunk = chunk[p + 1:]
> Just an idea. Not profiled.
I implemented this idea. It saves ~1 sec on perffncacheload for us when chunksize is taken 1 MB.
To: pulkit, #hg-reviewers
Cc: indygreg, yuja, mjpieters, mercurial-devel
More information about the Mercurial-devel