[PATCH] aggressivemergedeltas: enabled the option by default
Gregory Szorc
gregory.szorc at gmail.com
Tue Jul 10 01:29:23 EDT 2018
On Mon, Jul 9, 2018 at 9:55 AM, Gregory Szorc <gregory.szorc at gmail.com>
wrote:
> On Mon, Jul 9, 2018 at 6:35 AM, Boris Feld <boris.feld at octobus.net> wrote:
>
>> # HG changeset patch
>> # User Paul Morelle <paul.morelle at octobus.net>
>> # Date 1529624558 -7200
>> # Fri Jun 22 01:42:38 2018 +0200
>> # Node ID 799439bbfcc194238ff4af2979eb286c5defdc3f
>> # Parent a0e185f104541858a0b049e1fb67c4d113930a9a
>> # EXP-Topic enable-aggressivemergedeltas
>> # Available At https://bitbucket.org/octobus/mercurial-devel/
>> # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
>> 799439bbfcc1
>> aggressivemergedeltas: enabled the option by default
>>
>
> I'm +1 on enabling this feature by default: the performance improvements
> we've made to delta construction make the performance hit small enough that
> it justifies the size wins.
>
> However, I think this patch is too simple. Correct me if I'm wrong, but
> enabling format.aggressivemergedeltas will result in `hg unbundle`, `hg
> clone`, and `hg pull` computing deltas against incoming changesets (rather
> than using what was present in the bundle). This could add serious overhead
> to these operations.
>
> I think it is important that these operations remain fast. And that means
> they can't be recomputing deltas for incoming data.
>
> Could you please clarify the change in behavior for incoming bundle data?
> Perhaps you could include numbers for an `hg unbundle` on the measured
> repos?
>
> Also, when we enabled aggressivemergedeltas on mozilla-central, we had to
> set format.maxchainlen so delta chains didn't get too long. (I think some
> manifest chains grew to >40,000, which was very slow). I know there has
> been some changes to delta chain logic in the past year or two. Perhaps
> this is no longer an issue?
>
I compared `hg unbundle` performance for mozilla-unified before and after
this patch and didn't see any noticeable change in CPU consumption.
Honestly, this surprised me. I traced through the code and realized that
the interaction between finddeltainfo() and _getcandidaterevs() results in
the cached delta being used as long as it meats delta requirements (e.g.
delta chain length limits). In order to force delta computation, you need
to set format.generaldelta=true or not have a cached delta in play (e.g. in
the case of local commits).
So the remaining performance issue on the table is delta chain lengths. And
the commit message states this will be worked on in upcoming commits. So
please send those to the list ASAP so we can get them in 4.7.
Also, I think there's room to introduce config options to have better
control over delta generation. For example, I'd love if a server could
force delta computation on all incoming bundles (so a server ensures it
stores the smallest delta possible even if the push takes a little longer).
But we don't necessarily want to use format.generaldelta=true because then
any `hg pull` operations on that machine would result in delta computation.
i.e. I want a config option to control delta computation for a specific
operation/direction.
>
>
>>
>> The option has been around for a while (August 2015) but was never turned
>> on by
>> default. In-depth testing shows large wins for having that on with no
>> significant drawbacks.
>>
>> When enabled, revlog consider delta against both p1 and p2 at the same
>> time when
>> storing a revision. Selecting a delta against "p2" can produce better
>> deltas and
>> chain. This raise large benefit for all repositories, especially if they
>> have a
>> lot of merges.
>>
>> Comparison of `.hg/store/` size:
>>
>> mercurial (6.74% merges):
>> before: 54,225,348 bytes
>> after: 47,279,959 bytes -13%
>> pypy (8.30% merges):
>> before: 459,041,759 bytes
>> after: 346,090,067 bytes -25%
>> netbeans (34.21% merges):
>> before: 2,468,041,333 bytes
>> after: 1,364,077,645 bytes -45%
>> mozilla-central (4.84% merges):
>> before: 2,731,799,546 bytes
>> after: 2,157,718,019 bytes -21%
>>
>> Comparison of `00manifest.d` size:
>>
>> mercurial (6.74% merges):
>> before: 11,682,516 bytes
>> after: 6,143,044 bytes -47%
>> pypy (8.30% merges):
>> before: 156,447,163 bytes
>> after: 52,941,780 bytes -66%
>> netbeans (34.21% merges):
>> before: 1,250,363,851 bytes
>> after: 130,088,982 bytes -90%
>> mozilla-central (4.84% merges):
>> before: 468,202,733 bytes
>> after: 215,096,339 bytes -54%
>>
>> In addition, the better deltas help with the performance of multiple core
>> operations. However, better chains mean longer chains, which can affect
>> performance negatively (mostly manifest revision retrieval time). Chains
>> length
>> is a deeper problem that also affects linear repository too. Overall we
>> think
>> the benefits of using p2 as a diff target are bigger than the downsizes.
>> In
>> addition, we are also working on ways to improve the performance impact
>> of chain
>> length, so theses downsizes get fixed in the future.
>>
>> Below are interesting items from the full benchmark run:
>>
>> bundling 100 revisions from pypy:
>> before: 670ms
>> after: 480ms -28%
>> bundle 10000 revisions from pypy:
>> before: 1.38s
>> after: 1.10s -54%
>> bundle 10000 revisions from pypy:
>> before: 16.1s
>> after: 7.81s -52%
>> bundle 10000 revisions from netbeans:
>> before: 19.3s
>> after: 15.5s -19%
>>
>> unbundle 1000 revisions to pypy:
>> before: 641ms
>> after: 315ms - 51%
>>
>> clone mercurial (http):
>> before: 26.0s
>> after: 22.6s -23%
>>
>> pulling 1000 revisions from pypy (shh):
>> before: 2.07s
>> after: 1.36s -44%
>>
>> pushing 1000 revision through http (pypy repository)
>> before: 2.18s
>> after: 1.35s -48%
>>
>> diff time in mozilla-central:
>> before: 1.420s
>> after: 0.983s -31%
>>
>> status time in mozilla-central:
>> before: 1.260s
>> after: 0.828s -34%
>>
>> Impact in other cases seems minimal (within a couple of percent in worse
>> cases)
>> and can be seen in both direction:
>>
>> Timing for a simple `hg commit`:
>>
>> mozilla-central:
>> before: 3.37s
>> after: 3.22s -4%
>> pypy:
>> before: 194ms
>> after: 197ms +2%
>>
>> Timing for status (from tip to parent of tip):
>> mercurial:
>> before: 52.4ms
>> after: 52.4ms (same)
>> pypy:
>> before: 55.2
>> after: 56.9 +3%
>>
>> Timing for `hg update`
>>
>> mozilla-central, across 10 revisions:
>> before: 4.82s
>> after: 4.59s -5%
>> mozilla-central, across 10000 revisions:
>> before: 49.1s
>> after: 49.9s +2%
>> pypy, across 10 revisions:
>> before: 213ms
>> after: 216ms +1%
>> pypy, across 10000 revisions:
>> before: 5.31ms
>> after: 5.24ms -1%
>>
>> The negative consequences are related to manifest fetch time:
>> (timing for the tip revision tested by the benchmark)
>>
>> pypy-2018:
>> before: 2.60ms
>> after: 3.88ms +50%
>> mozilla-central-2018:
>> before: 565ms
>> after: 652ms +15% (~+100ms)
>> netbeans-2018:
>> before: 101ms
>> after: 250ms +48% (~+150ms)
>>
>> This shows up as a fixed overhead on some command we benchmarked:
>>
>> no-op push of mozilla-central:
>> before: 945ms
>> after: 1040ms +10% (~+100ms)
>> pushing 10 changeset in netbeabs over ssh:
>> before: 557ms
>> after: 712ms +28% (+155ms)
>> pushing 100 changeset in netbeabs over ssh:
>> before: 592ms
>> after: 771ms +30% (+179ms)
>>
>> diff --git a/mercurial/configitems.py b/mercurial/configitems.py
>> --- a/mercurial/configitems.py
>> +++ b/mercurial/configitems.py
>> @@ -617,7 +617,7 @@ coreconfigitem('extdata', '.*',
>> generic=True,
>> )
>> coreconfigitem('format', 'aggressivemergedeltas',
>> - default=False,
>> + default=True,
>> )
>> coreconfigitem('format', 'chunkcachesize',
>> default=None,
>> diff --git a/mercurial/revlog.py b/mercurial/revlog.py
>> --- a/mercurial/revlog.py
>> +++ b/mercurial/revlog.py
>> @@ -615,7 +615,7 @@ class revlog(object):
>> # How much data to read and cache into the raw revlog data cache.
>> self._chunkcachesize = 65536
>> self._maxchainlen = None
>> - self._aggressivemergedeltas = False
>> + self._aggressivemergedeltas = True
>> self.index = []
>> # Mapping of partial identifiers to full nodes.
>> self._pcache = {}
>> diff --git a/tests/test-generaldelta.t b/tests/test-generaldelta.t
>> --- a/tests/test-generaldelta.t
>> +++ b/tests/test-generaldelta.t
>> @@ -139,7 +139,7 @@ Test format.aggressivemergedeltas
>> rev chain# chainlen prev delta size rawsize
>> chainsize ratio lindist extradist extraratio
>> 0 1 1 -1 base 59 215
>> 59 0.27442 59 0 0.00000
>> 1 1 2 0 prev 61 86
>> 120 1.39535 120 0 0.00000
>> - 2 1 3 1 p1 65 301
>> 185 0.61462 185 0 0.00000
>> + 2 1 2 0 p2 62 301
>> 121 0.40199 182 61 0.50413
>>
>> $ hg strip -q -r . --config extensions.strip=
>>
>> _______________________________________________
>> Mercurial-devel mailing list
>> Mercurial-devel at mercurial-scm.org
>> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20180709/44f74551/attachment.html>
More information about the Mercurial-devel
mailing list