[PATCH 3 of 3] use per-directory clustered stat calls even in cases where known tree is walked

Benoit Boissinot bboissin at gmail.com
Wed Oct 1 04:45:14 CDT 2008


On Wed, Oct 01, 2008 at 11:21:53AM +0200, Benoit Boissinot wrote:
> On Wed, Oct 01, 2008 at 10:51:34AM +0200, Christian Boos wrote:
> > Benoit Boissinot wrote:
> >> On Tue, Sep 30, 2008 at 10:48:29PM -0400, Petr Kodl wrote:
> >> ...
> >> I was thinking something more like this, could you test on windows?
> >> (the testsuite runs ok on linux)
> >> - some doc needs be added to statfiles()
> >> - the correct exception (ENOENT?) should be catched when listdir fails
> >>
> >> diff --git a/mercurial/dirstate.py b/mercurial/dirstate.p
> > (snip)
> >
> > Just a quick note to mention that the patch seemed to works fine - at  
> > first.
> 
> Ok I know what went wrong.
> 
> Please change:
>     ls = dict([(f.lower(), st) for
>                f, st in osutil.listdir(dir, stat=True)])
> except:
> 
> to (listdir returns a triplet, not a couple):
>     ls = dict([(f.lower(), st) for
>                f, kind, st in osutil.listdir(dir, stat=True)])
> except OSError:
> 
> I really shouldn't have catched all errors.

There was another problem (it should remove the '/' when splitting, maybe I could
add strutil.rsplit()).

The following is better tested (I tried the same function minus the .lower() calls
on linux):

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -532,17 +532,13 @@
                         results[nf] = None
 
         # step 3: report unseen items in the dmap hash
-        visit = [f for f in dmap if f not in results and match(f)]
-        for nf in util.sort(visit):
-            results[nf] = None
-            try:
-                st = lstat(join(nf))
+        visit = util.sort([f for f in dmap if f not in results and match(f)])
+        for nf, st in zip(visit, util.statfiles([join(f) for f in visit])):
+            if st is not None:
                 kind = getkind(st.st_mode)
-                if kind == regkind or kind == lnkkind:
-                    results[nf] = st
-            except OSError, inst:
-                if inst.errno not in (errno.ENOENT, errno.ENOTDIR):
-                    raise
+                if kind != regkind and kind != lnkkind:
+                    st = None
+            results[nf] = st
 
         del results['.hg']
         return results
diff --git a/mercurial/util.py b/mercurial/util.py
--- a/mercurial/util.py
+++ b/mercurial/util.py
@@ -1162,6 +1162,25 @@
         except NameError:
             pass
 
+    def statfiles(files):
+        dircache = {}
+        for fn in files:
+            pos = fn.rfind('/')
+            if pos != -1:
+                dir, base = fn[:pos].lower(), fn[pos+1:].lower()
+            else:
+                dir, base = '', fn.lower()
+            try:
+                ls = dircache[dir]
+            except KeyError:
+                try:
+                    ls = dict([(f.lower(), st) for
+                               f, kind, st in osutil.listdir(dir, stat=True)])
+                except OSError:
+                    ls = {}
+                dircache[dir] = ls
+            yield ls.get(base, None)
+
     try:
         # override functions with win32 versions if possible
         from util_win32 import *
@@ -1334,6 +1353,15 @@
 
     def set_signal_handler():
         pass
+
+    def statfiles(files):
+        for fn in files:
+            try:
+                yield os.lstat(fn)
+            except OSError, inst:
+                if inst.errno not in (errno.ENOENT, errno.ENOTDIR):
+                    raise
+                yield None
 
 def find_exe(name, default=None):
     '''find path of an executable.

-- 
:wq


More information about the Mercurial-devel mailing list