[PATCH] aggressivemergedeltas: enabled the option by default
Gregory Szorc
gregory.szorc at gmail.com
Mon Jul 9 12:55:13 EDT 2018
On Mon, Jul 9, 2018 at 6:35 AM, Boris Feld <boris.feld at octobus.net> wrote:
> # HG changeset patch
> # User Paul Morelle <paul.morelle at octobus.net>
> # Date 1529624558 -7200
> # Fri Jun 22 01:42:38 2018 +0200
> # Node ID 799439bbfcc194238ff4af2979eb286c5defdc3f
> # Parent a0e185f104541858a0b049e1fb67c4d113930a9a
> # EXP-Topic enable-aggressivemergedeltas
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
> 799439bbfcc1
> aggressivemergedeltas: enabled the option by default
>
I'm +1 on enabling this feature by default: the performance improvements
we've made to delta construction make the performance hit small enough that
it justifies the size wins.
However, I think this patch is too simple. Correct me if I'm wrong, but
enabling format.aggressivemergedeltas will result in `hg unbundle`, `hg
clone`, and `hg pull` computing deltas against incoming changesets (rather
than using what was present in the bundle). This could add serious overhead
to these operations.
I think it is important that these operations remain fast. And that means
they can't be recomputing deltas for incoming data.
Could you please clarify the change in behavior for incoming bundle data?
Perhaps you could include numbers for an `hg unbundle` on the measured
repos?
Also, when we enabled aggressivemergedeltas on mozilla-central, we had to
set format.maxchainlen so delta chains didn't get too long. (I think some
manifest chains grew to >40,000, which was very slow). I know there has
been some changes to delta chain logic in the past year or two. Perhaps
this is no longer an issue?
>
> The option has been around for a while (August 2015) but was never turned
> on by
> default. In-depth testing shows large wins for having that on with no
> significant drawbacks.
>
> When enabled, revlog consider delta against both p1 and p2 at the same
> time when
> storing a revision. Selecting a delta against "p2" can produce better
> deltas and
> chain. This raise large benefit for all repositories, especially if they
> have a
> lot of merges.
>
> Comparison of `.hg/store/` size:
>
> mercurial (6.74% merges):
> before: 54,225,348 bytes
> after: 47,279,959 bytes -13%
> pypy (8.30% merges):
> before: 459,041,759 bytes
> after: 346,090,067 bytes -25%
> netbeans (34.21% merges):
> before: 2,468,041,333 bytes
> after: 1,364,077,645 bytes -45%
> mozilla-central (4.84% merges):
> before: 2,731,799,546 bytes
> after: 2,157,718,019 bytes -21%
>
> Comparison of `00manifest.d` size:
>
> mercurial (6.74% merges):
> before: 11,682,516 bytes
> after: 6,143,044 bytes -47%
> pypy (8.30% merges):
> before: 156,447,163 bytes
> after: 52,941,780 bytes -66%
> netbeans (34.21% merges):
> before: 1,250,363,851 bytes
> after: 130,088,982 bytes -90%
> mozilla-central (4.84% merges):
> before: 468,202,733 bytes
> after: 215,096,339 bytes -54%
>
> In addition, the better deltas help with the performance of multiple core
> operations. However, better chains mean longer chains, which can affect
> performance negatively (mostly manifest revision retrieval time). Chains
> length
> is a deeper problem that also affects linear repository too. Overall we
> think
> the benefits of using p2 as a diff target are bigger than the downsizes. In
> addition, we are also working on ways to improve the performance impact of
> chain
> length, so theses downsizes get fixed in the future.
>
> Below are interesting items from the full benchmark run:
>
> bundling 100 revisions from pypy:
> before: 670ms
> after: 480ms -28%
> bundle 10000 revisions from pypy:
> before: 1.38s
> after: 1.10s -54%
> bundle 10000 revisions from pypy:
> before: 16.1s
> after: 7.81s -52%
> bundle 10000 revisions from netbeans:
> before: 19.3s
> after: 15.5s -19%
>
> unbundle 1000 revisions to pypy:
> before: 641ms
> after: 315ms - 51%
>
> clone mercurial (http):
> before: 26.0s
> after: 22.6s -23%
>
> pulling 1000 revisions from pypy (shh):
> before: 2.07s
> after: 1.36s -44%
>
> pushing 1000 revision through http (pypy repository)
> before: 2.18s
> after: 1.35s -48%
>
> diff time in mozilla-central:
> before: 1.420s
> after: 0.983s -31%
>
> status time in mozilla-central:
> before: 1.260s
> after: 0.828s -34%
>
> Impact in other cases seems minimal (within a couple of percent in worse
> cases)
> and can be seen in both direction:
>
> Timing for a simple `hg commit`:
>
> mozilla-central:
> before: 3.37s
> after: 3.22s -4%
> pypy:
> before: 194ms
> after: 197ms +2%
>
> Timing for status (from tip to parent of tip):
> mercurial:
> before: 52.4ms
> after: 52.4ms (same)
> pypy:
> before: 55.2
> after: 56.9 +3%
>
> Timing for `hg update`
>
> mozilla-central, across 10 revisions:
> before: 4.82s
> after: 4.59s -5%
> mozilla-central, across 10000 revisions:
> before: 49.1s
> after: 49.9s +2%
> pypy, across 10 revisions:
> before: 213ms
> after: 216ms +1%
> pypy, across 10000 revisions:
> before: 5.31ms
> after: 5.24ms -1%
>
> The negative consequences are related to manifest fetch time:
> (timing for the tip revision tested by the benchmark)
>
> pypy-2018:
> before: 2.60ms
> after: 3.88ms +50%
> mozilla-central-2018:
> before: 565ms
> after: 652ms +15% (~+100ms)
> netbeans-2018:
> before: 101ms
> after: 250ms +48% (~+150ms)
>
> This shows up as a fixed overhead on some command we benchmarked:
>
> no-op push of mozilla-central:
> before: 945ms
> after: 1040ms +10% (~+100ms)
> pushing 10 changeset in netbeabs over ssh:
> before: 557ms
> after: 712ms +28% (+155ms)
> pushing 100 changeset in netbeabs over ssh:
> before: 592ms
> after: 771ms +30% (+179ms)
>
> diff --git a/mercurial/configitems.py b/mercurial/configitems.py
> --- a/mercurial/configitems.py
> +++ b/mercurial/configitems.py
> @@ -617,7 +617,7 @@ coreconfigitem('extdata', '.*',
> generic=True,
> )
> coreconfigitem('format', 'aggressivemergedeltas',
> - default=False,
> + default=True,
> )
> coreconfigitem('format', 'chunkcachesize',
> default=None,
> diff --git a/mercurial/revlog.py b/mercurial/revlog.py
> --- a/mercurial/revlog.py
> +++ b/mercurial/revlog.py
> @@ -615,7 +615,7 @@ class revlog(object):
> # How much data to read and cache into the raw revlog data cache.
> self._chunkcachesize = 65536
> self._maxchainlen = None
> - self._aggressivemergedeltas = False
> + self._aggressivemergedeltas = True
> self.index = []
> # Mapping of partial identifiers to full nodes.
> self._pcache = {}
> diff --git a/tests/test-generaldelta.t b/tests/test-generaldelta.t
> --- a/tests/test-generaldelta.t
> +++ b/tests/test-generaldelta.t
> @@ -139,7 +139,7 @@ Test format.aggressivemergedeltas
> rev chain# chainlen prev delta size rawsize
> chainsize ratio lindist extradist extraratio
> 0 1 1 -1 base 59 215
> 59 0.27442 59 0 0.00000
> 1 1 2 0 prev 61 86
> 120 1.39535 120 0 0.00000
> - 2 1 3 1 p1 65 301
> 185 0.61462 185 0 0.00000
> + 2 1 2 0 p2 62 301
> 121 0.40199 182 61 0.50413
>
> $ hg strip -q -r . --config extensions.strip=
>
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20180709/a16faab2/attachment.html>
More information about the Mercurial-devel
mailing list