[PATCH 1 of 4] perf: add command for measuring revlog chunk operations

Gregory Szorc gregory.szorc at gmail.com
Tue Nov 1 21:16:36 EDT 2016


# HG changeset patch
# User Gregory Szorc <gregory.szorc at gmail.com>
# Date 1478048814 25200
#      Tue Nov 01 18:06:54 2016 -0700
# Node ID fb93d9a0a24db5a93a6a6758eacc6ba5ca37531e
# Parent  e5cc44ea12de681d971fcbebb65a7fb71fd1c3c7
perf: add command for measuring revlog chunk operations

Upcoming commits will refactor the revlog compression APIs in
preparation for having better support for new compression engines.
We want to be sure this refactoring doesn't regress performance.
So let's introduce "perfrevchunks" to explicitly test performance of
reading, decompressing, and recompressing revlog chunks.

Here is example output when run on the mozilla-unified repo:

$ hg perfrevlogchunks -c
! read
! wall 0.538873 comb 0.530000 user 0.490000 sys 0.040000 (best of 19)
! read w/ reused fd
! wall 0.523037 comb 0.520000 user 0.500000 sys 0.020000 (best of 19)
! read batch
! wall 0.013405 comb 0.010000 user 0.000000 sys 0.010000 (best of 217)
! read batch w/ reused fd
! wall 0.013448 comb 0.020000 user 0.000000 sys 0.020000 (best of 217)
! chunk
! wall 2.114581 comb 2.110000 user 2.080000 sys 0.030000 (best of 5)
! chunk batch
! wall 1.741836 comb 1.740000 user 1.710000 sys 0.030000 (best of 6)
! compress
! wall 5.471946 comb 5.480000 user 5.480000 sys 0.000000 (best of 3)

$ hg perfrevlogchunks -m
! read
! wall 0.557251 comb 0.560000 user 0.530000 sys 0.030000 (best of 18)
! read w/ reused fd
! wall 0.536264 comb 0.550000 user 0.500000 sys 0.050000 (best of 19)
! read batch
! wall 0.027415 comb 0.030000 user 0.000000 sys 0.030000 (best of 107)
! read batch w/ reused fd
! wall 0.027409 comb 0.030000 user 0.000000 sys 0.030000 (best of 108)
! chunk
! wall 2.696220 comb 2.700000 user 2.650000 sys 0.050000 (best of 4)
! chunk batch
! wall 2.342927 comb 2.350000 user 2.320000 sys 0.030000 (best of 5)
! compress
! wall 9.640969 comb 9.650000 user 9.650000 sys 0.000000 (best of 3)

We already see some interesting data, such as how much slower
non-batched chunk reading is and that zlib compression appears to be
>2x slower than decompression.

The tests for file descriptor reuse don't seem interesting on this
machine. However, when testing on a repo stored on an NFS volume,
file descriptor reuse does matter. So I think there is value to
measuring this.

diff --git a/contrib/perf.py b/contrib/perf.py
--- a/contrib/perf.py
+++ b/contrib/perf.py
@@ -799,6 +799,84 @@ def perfrevlog(ui, repo, file_=None, sta
     timer(d)
     fm.end()
 
+ at command('perfrevlogchunks', revlogopts + formatteropts +
+         [('s', 'startrev', 0, 'revision to start at')],
+         '-c|-m|FILE')
+def perfrevlogchunks(ui, repo, file_=None, startrev=0, **opts):
+    """Benchmark operations on revlog chunks.
+
+    Logically, each revlog is a collection of fulltext revisions. However,
+    stored within each revlog are "chunks" of possibly compressed data. This
+    data needs to be read and decompressed or compressed and written.
+
+    This command measures the time it takes to read+decompress and recompress
+    chunks in a revlog. It effectively isolates I/O and compression performance.
+    For measurements of higher-level operations like resolving revisions,
+    see ``perfrevlog`` and ``perfrevlogrevision``.
+    """
+    rl = cmdutil.openrevlog(repo, 'perfrevlogchunks', file_, opts)
+    revs = list(rl.revs(startrev, len(rl) - 1))
+
+    def rlfh(rl):
+        if rl._inline:
+            return getsvfs(repo)(rl.indexfile)
+        else:
+            return getsvfs(repo)(rl.datafile)
+
+    def doread():
+        rl.clearcaches()
+        for rev in revs:
+            rl._chunkraw(rev, rev)
+
+    def doreadcachedfh():
+        rl.clearcaches()
+        fh = rlfh(rl)
+        for rev in revs:
+            rl._chunkraw(rev, rev, df=fh)
+
+    def doreadbatch():
+        rl.clearcaches()
+        rl._chunkraw(revs[0], revs[-1])
+
+    def doreadbatchcachedfh():
+        rl.clearcaches()
+        fh = rlfh(rl)
+        rl._chunkraw(revs[0], revs[-1], df=fh)
+
+    def dochunk():
+        rl.clearcaches()
+        fh = rlfh(rl)
+        for rev in revs:
+            rl._chunk(rev, df=fh)
+
+    chunks = [None]
+
+    def dochunkbatch():
+        rl.clearcaches()
+        fh = rlfh(rl)
+        # Save chunks as a side-effect.
+        chunks[0] = rl._chunks(revs, df=fh)
+
+    def docompress():
+        rl.clearcaches()
+        for chunk in chunks[0]:
+            rl.compress(chunk)
+
+    benches = [
+        (lambda: doread(), 'read'),
+        (lambda: doreadcachedfh(), 'read w/ reused fd'),
+        (lambda: doreadbatch(), 'read batch'),
+        (lambda: doreadbatchcachedfh(), 'read batch w/ reused fd'),
+        (lambda: dochunk(), 'chunk'),
+        (lambda: dochunkbatch(), 'chunk batch'),
+        (lambda: docompress(), 'compress'),
+    ]
+
+    for fn, title in benches:
+        timer, fm = gettimer(ui, opts)
+        timer(fn, title=title)
+        fm.end()
+
 @command('perfrevlogrevision', revlogopts + formatteropts +
          [('', 'cache', False, 'use caches instead of clearing')],
          '-c|-m|FILE REV')
diff --git a/tests/test-contrib-perf.t b/tests/test-contrib-perf.t
--- a/tests/test-contrib-perf.t
+++ b/tests/test-contrib-perf.t
@@ -94,6 +94,8 @@ perfstatus
                  (no help text available)
    perfrawfiles  (no help text available)
    perfrevlog    Benchmark reading a series of revisions from a revlog.
+   perfrevlogchunks
+                 Benchmark operations on revlog chunks.
    perfrevlogrevision
                  Benchmark obtaining a revlog revision.
    perfrevrange  (no help text available)
@@ -141,6 +143,7 @@ perfstatus
   $ hg perfrawfiles 2
   $ hg perfrevlog .hg/store/data/a.i
   $ hg perfrevlogrevision -m 0
+  $ hg perfrevlogchunks -c
   $ hg perfrevrange
   $ hg perfrevset 'all()'
   $ hg perfstartup


More information about the Mercurial-devel mailing list