D5296: store: don't read the whole fncache in memory

Yuya Nishihara yuya at tcha.org
Mon Feb 25 22:01:15 EST 2019


(resend without the "On ... wrote:" line)

>   Seeing the performance benefit it brings on our repo, I want to try other ways we can do this. Do we like having a conditional which checks the size of fncache and choose one of the approaches depending on how large it is?
>   
>   If size(fncache) < 50M, use the current approach
>   else, use the approach described in this patch

Suppose the current code is fast because it avoids running Python in loop,
I think it can be extended to not use too much memory.

```
chunk = b''
while True:
    chunk += fp.read(chunk_size)  # maybe ~10MB?
    p = chunk.rindex(b'\n')  # need to handle error
    decodedir(chunk[:p + 1])...
    chunk = chunk[p + 1:]
```

Just an idea. Not profiled.


More information about the Mercurial-devel mailing list