D5296: store: don't read the whole fncache in memory

[pulkit (Pulkit Goyal)]

pulkit added a comment.


  In https://phab.mercurial-scm.org/D5296#87892, @yuja wrote:
  
  > (resend without the "On ... wrote:" line)
  >
  > >   Seeing the performance benefit it brings on our repo, I want to try other ways we can do this. Do we like having a conditional which checks the size of fncache and choose one of the approaches depending on how large it is?
  > >   
  > >   If size(fncache) < 50M, use the current approach
  > >   else, use the approach described in this patch
  >
  > Suppose the current code is fast because it avoids running Python in loop,
  >  I think it can be extended to not use too much memory.
  >
  >   chunk = b''
  >   while True:
  >       chunk += fp.read(chunk_size)  # maybe ~10MB?
  >       p = chunk.rindex(b'\n')  # need to handle error
  >       decodedir(chunk[:p + 1])...
  >       chunk = chunk[p + 1:]
  >
  >
  > Just an idea. Not profiled.
  
  
  I implemented this idea. It saves ~1 sec on perffncacheload for us when chunksize is taken 1 MB.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D5296

To: pulkit, #hg-reviewers
Cc: indygreg, yuja, mjpieters, mercurial-devel