[PATCH 2 of 2] perf: benchmark command for revlog indexes

[Pierre-Yves David]

On 05/27/2017 05:41 AM, Gregory Szorc wrote:
> # HG changeset patch
> # User Gregory Szorc <gregory.szorc at gmail.com>
> # Date 1495856487 25200
> #      Fri May 26 20:41:27 2017 -0700
> # Node ID 84ec213e509859c3d983ba537b25a025d61f31d9
> # Parent  684fbf0453f4abc3c2ccabc05cef3f4233d6476c
> perf: benchmark command for revlog indexes

Looks good and works good, Thanks! I've marked it as pre-reviewed.
I've added a small python 3 nits below.

> We didn't have explicit microbenchmark coverage for loading revlog
> indexes. That seems like a useful thing to have, so let's add it.
>
> We currently measure the low-level nodemap APIs. There is room to
> hook in at the actual revlog layer. This could be done as a follow-up.
>
> The hackiest thing about this patch is specifying revlog paths.
> Other commands have arguments that allow resolution of changelog,
> manifest, and filelog. I needed to hook in at a lower level of
> the revlog API than what the existing helper functions to resolve
> revlogs allowed. I was too lazy to write some new APIs. This could
> be done as a follow-up easily enough.
>
> Example output for `hg perfrevlogindex 00changelog.i` on my
> Firefox repo (404418 revisions):
>
> ! revlog constructor
> ! wall 0.003106 comb 0.000000 user 0.000000 sys 0.000000 (best of 912)
> ! read
> ! wall 0.003077 comb 0.000000 user 0.000000 sys 0.000000 (best of 924)
> ! create index object
> ! wall 0.000000 comb 0.000000 user 0.000000 sys 0.000000 (best of 1803994)
> ! retrieve index entry for rev 0
> ! wall 0.000193 comb 0.000000 user 0.000000 sys 0.000000 (best of 14037)
> ! look up missing node
> ! wall 0.003313 comb 0.000000 user 0.000000 sys 0.000000 (best of 865)
> ! look up node at rev 0
> ! wall 0.003295 comb 0.010000 user 0.010000 sys 0.000000 (best of 858)
> ! look up node at 1/4 len
> ! wall 0.002598 comb 0.010000 user 0.010000 sys 0.000000 (best of 1103)
> ! look up node at 1/2 len
> ! wall 0.001909 comb 0.000000 user 0.000000 sys 0.000000 (best of 1507)
> ! look up node at 3/4 len
> ! wall 0.001213 comb 0.000000 user 0.000000 sys 0.000000 (best of 2275)
> ! look up node at tip
> ! wall 0.000453 comb 0.000000 user 0.000000 sys 0.000000 (best of 5697)
> ! look up all nodes (forward)
> ! wall 0.094615 comb 0.100000 user 0.100000 sys 0.000000 (best of 100)
> ! look up all nodes (reverse)
> ! wall 0.045889 comb 0.050000 user 0.050000 sys 0.000000 (best of 100)
> ! retrieve all index entries (forward)
> ! wall 0.078398 comb 0.080000 user 0.060000 sys 0.020000 (best of 100)
> ! retrieve all index entries (reverse)
> ! wall 0.079376 comb 0.080000 user 0.070000 sys 0.010000 (best of 100)
>
> diff --git a/contrib/perf.py b/contrib/perf.py
> --- a/contrib/perf.py
> +++ b/contrib/perf.py
> @@ -23,6 +23,7 @@ import functools
>  import gc
>  import os
>  import random
> +import struct
>  import sys
>  import time
>  from mercurial import (
> @@ -34,6 +35,7 @@ from mercurial import (
>      extensions,
>      mdiff,
>      merge,
> +    revlog,
>      util,
>  )
>
> @@ -61,6 +63,10 @@ try:
>      from mercurial import scmutil # since 1.9 (or 8b252e826c68)
>  except ImportError:
>      pass
> +try:
> +    from mercurial import pycompat # since 3.8 (or 6041fb8f2da8)
> +except ImportError:
> +    pass
>
>  # for "historical portability":
>  # define util.safehasattr forcibly, because util.safehasattr has been
> @@ -857,6 +863,106 @@ def perfdiffwd(ui, repo, **opts):
>          timer(d, title)
>      fm.end()
>
> + at command('perfrevlogindex', formatteropts, 'FILE')
> +def perfrevlogindex(ui, repo, file_, **opts):
> +    """Benchmark operations against a revlog index.
> +
> +    This tests constructing a revlog instance, reading index data,
> +    parsing index data, and performing various operations related to
> +    index data.
> +    """
> +
> +    opener = getsvfs(repo)
> +
> +    data = opener.read(file_)
> +
> +    header = struct.unpack('>I', data[0:4])[0]
> +    version = header & 0xFFFF
> +    if version == 1:
> +        revlogio = revlog.revlogio()
> +        inline = header & (1 << 16)
> +    else:
> +        raise error.Abort(_('unsupported revlog version: %d') % version)
> +
> +    rl = revlog.revlog(opener, file_)
> +    rllen = len(rl)
> +
> +    node0 = rl.node(0)
> +    node25 = rl.node(rllen / 4)
> +    node50 = rl.node(rllen / 2)
> +    node75 = rl.node(rllen / 4 * 3)

note, you want these to be "//" instead of "/" or python 3 will get you.

Cheers,

-- 
Pierre-Yves David