D5296: store: don't read the whole fncache in memory

[pulkit (Pulkit Goyal)]

pulkit updated this revision to Diff 14256.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D5296?vs=14255&id=14256

REVISION DETAIL
  https://phab.mercurial-scm.org/D5296

AFFECTED FILES
  mercurial/store.py

CHANGE DETAILS

diff --git a/mercurial/store.py b/mercurial/store.py
--- a/mercurial/store.py
+++ b/mercurial/store.py
@@ -463,7 +463,27 @@
             # skip nonexistent file
             self.entries = set()
             return
-        self.entries = set(decodedir(fp.read()).splitlines())
+
+        self.entries = []
+        totalsize = self.vfs.stat('fncache').st_size
+        chunksize = (10 ** 6) # 1 MB
+        chunk = b''
+        chunksize = min(totalsize, chunksize)
+        totalsize -= chunksize
+        while chunksize:
+            chunk += fp.read(chunksize)
+            try:
+                p = chunk.rindex(b'\n')
+                self.entries.extend(decodedir(chunk[:p + 1]).splitlines())
+                chunk = chunk[p + 1:]
+            except ValueError:
+                # substring '\n' not found, maybe the entry is bigger than the
+                # chunksize, so let's keep iterating
+                pass
+            chunksize = min(totalsize, chunksize)
+            totalsize -= chunksize
+
+        self.entries = set(self.entries)
         self._checkentries(fp)
         fp.close()
 



To: pulkit, #hg-reviewers
Cc: indygreg, yuja, mjpieters, mercurial-devel