D5296: store: don't read the whole fncache in memory

[pulkit (Pulkit Goyal)]

pulkit created this revision.
Herald added subscribers: mercurial-devel, mjpieters.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  In large repositories with lot of files, the fncache grows more than 100 MB and
  reading that whole thing into memory slows things down. Let's not read the whole
  thing into memory.
  
  Following all the performance optimizations observed:
  
  `hg perffncacheload` (best of 4 runs)
  
  before: wall 3.253480 comb 3.440000 user 2.500000 sys 0.940000 (best of 3)
  after: wall 2.850717 comb 3.010000 user 2.370000 sys 0.640000 (best of 4)

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D5296

AFFECTED FILES
  mercurial/store.py

CHANGE DETAILS

diff --git a/mercurial/store.py b/mercurial/store.py
--- a/mercurial/store.py
+++ b/mercurial/store.py
@@ -461,13 +461,13 @@
             # skip nonexistent file
             self.entries = set()
             return
-        self.entries = set(decodedir(fp.read()).splitlines())
-        if '' in self.entries:
-            fp.seek(0)
-            for n, line in enumerate(util.iterfile(fp)):
-                if not line.rstrip('\n'):
-                    t = _('invalid entry in fncache, line %d') % (n + 1)
-                    raise error.Abort(t)
+        self.entries = set()
+        for n, line in enumerate(util.iterfile(fp)):
+            entry = line.rstrip('\n')
+            if not entry:
+                t = _('invalid entry in fncache, line %d') % (n + 1)
+                raise error.Abort(t)
+            self.entries.add(decodedir(entry))
         fp.close()
 
     def write(self, tr):



To: pulkit, #hg-reviewers
Cc: mjpieters, mercurial-devel