[PATCH 2 of 2] treemanifest: speed up 4x tree to flat manifest conversion

Laurent Charignon lcharignon at fb.com
Thu Feb 4 14:38:45 EST 2016


We don't currently have commands relying directly on that but I assume that it will be useful if we want to have a server with tree manifest supporting old clients with flat manifest.
I was just playing around with tree manifest doing some conversions and noticed that it was taking a very long time to call that method.
The original code was going through the tree manifest hierarchy 3 times(filename, flag and node) when only once was needed: it was a low hanging fruit for performance.

Laurent

On Feb 4, 2016, at 10:43 AM, Martin von Zweigbergk <martinvonz at google.com<mailto:martinvonz at google.com>> wrote:

And why is this important? I know many others like the perf commands, but I prefer to know what commands it speeds up. Could you please people like me too by including timings for an end-user command?

On Thu, Feb 4, 2016, 10:04 Laurent Charignon <lcharignon at fb.com<mailto:lcharignon at fb.com>> wrote:
# HG changeset patch
# User Laurent Charignon <lcharignon at fb.com<mailto:lcharignon at fb.com>>
# Date 1454606782 28800
#      Thu Feb 04 09:26:22 2016 -0800
# Branch stable
# Node ID b163d0993d788fb7c5a55a8a052b1115ba7b66e6
# Parent  241c38b8b3acbf8d2ab57fa85a38e337ef4e2472
treemanifest: speed up 4x tree to flat manifest conversion

On our big repos we have a 4x speed improvement in tree to flat manifest
conversion:

Before this patch
~/hg/hg perftreemanifesttotext
! wall 25.622450 comb 25.590000 user 25.410000 sys 0.180000 (best of 3)

After this patch
~/hg/hg perftreemanifesttotext
! wall 6.541120 comb 6.540000 user 6.470000 sys 0.070000 (best of 3)

diff --git a/mercurial/manifest.py b/mercurial/manifest.py
--- a/mercurial/manifest.py
+++ b/mercurial/manifest.py
@@ -848,10 +848,20 @@

     def text(self, usemanifestv2=False):
         """Get the full data of this manifest as a bytestring."""
-        self._load()
-        flags = self.flags
-        return _text(((f, self[f], flags(f)) for f in self.keys()),
-                     usemanifestv2)
+        tovisit = []
+        visit = tovisit.pop
+        tovisit.append(self)
+        res = []
+
+        while tovisit:
+            n = visit()
+            n._load()
+            tovisit.extend(n._dirs.values())
+            dirpath = n._dir
+            getflags = n._flags.get
+            res.extend([(dirpath + f, u, getflags(f, ''))
+                       for (f, u) in n._files.items()])
+        return _text(sorted(res), usemanifestv2)

     def dirtext(self, usemanifestv2=False):
         """Get the full data of this directory as a bytestring. Make sure that
_______________________________________________
Mercurial-devel mailing list
Mercurial-devel at mercurial-scm.org<mailto:Mercurial-devel at mercurial-scm.org>
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mercurial-2Dscm.org_mailman_listinfo_mercurial-2Ddevel&d=CwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=qmwlQ6ljsf0--v3ANP53-V-RM6PPUtJ5zK5Y1fStJGg&m=tdWWjosTIDQbppobtsBZ4R7iO15aQkModRL5ZRdUIb0&s=xzaT80hvmCsFEGxrew7c7me3uZpu-vNyxWPrQen3zo8&e=>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160204/907e00ae/attachment.html>


More information about the Mercurial-devel mailing list