D5296: store: don't read the whole fncache in memory

[yuja (Yuya Nishihara)]

yuja added a comment.


  (resend without the "On ... wrote:" line)
  
  >   Seeing the performance benefit it brings on our repo, I want to try other ways we can do this. Do we like having a conditional which checks the size of fncache and choose one of the approaches depending on how large it is?
  >   
  >   If size(fncache) < 50M, use the current approach
  >   else, use the approach described in this patch
  
  Suppose the current code is fast because it avoids running Python in loop,
  I think it can be extended to not use too much memory.
  
    chunk = b''
    while True:
        chunk += fp.read(chunk_size)  # maybe ~10MB?
        p = chunk.rindex(b'\n')  # need to handle error
        decodedir(chunk[:p + 1])...
        chunk = chunk[p + 1:]
  
  Just an idea. Not profiled.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D5296

To: pulkit, #hg-reviewers
Cc: indygreg, yuja, mjpieters, mercurial-devel