[PATCH 1 of 5 rfc] tests: explore some bdiff cases

[Gregory Szorc]

On Mon, Nov 7, 2016 at 6:47 AM, Pierre-Yves David <
pierre-yves.david at ens-lyon.org> wrote:

>
>
> On 11/06/2016 04:56 PM, Mads Kiilerich wrote:
>
>> On 11/06/2016 10:07 AM, Yuya Nishihara wrote:
>>
>>> On Thu, 03 Nov 2016 22:34:11 +0100, Mads Kiilerich wrote:
>>>
>>>> # HG changeset patch
>>>> # User Mads Kiilerich <madski at unity3d.com>
>>>> # Date 1478208837 -3600
>>>> #      Thu Nov 03 22:33:57 2016 +0100
>>>> # Node ID f6408efe0d0f4179fe6cc2b967164c1b4567f3d6
>>>> # Parent  d06c049695e6ad3219e7479c65ce98a2f123e878
>>>> tests: explore some bdiff cases
>>>>
>>>> diff --git a/tests/test-bhalf.t b/tests/test-bhalf.t
>>>> new file mode 100644
>>>> --- /dev/null
>>>> +++ b/tests/test-bhalf.t
>>>>
>>> '#require no-pure' is necessary since we use difflib in pure.
>>>
>>> The other changes in this series look good to me, but it's bdiff.c so I
>>> don't queue them.
>>>
>>
>> Thanks for reviewing and the positive feedback. I will try to polish it
>> for "real" submission.
>>
>
> Like yuya, it does not seems to be anything wrong in this series (and it
> seems to move in the right direction). That last last statement from you is
> bit confusing to me as the series seems okay for inclusion but you seems to
> imply you'll resent it.
>
> I'm also adding Jun and Greg in CC as they have been looking into bdiff
> recently and I'm curious about what they might think.
>
> Also, it might be a good idea to have some timing data in the descript to
> show that the new code is not making things slower (but we really should
> just have some automated performance tracking for this at this point :-/)
>
> For the last patch, I wonder if it would be better to add a post
>> processing step that - given all the chunks - try to shift/rotate all
>> match sequences to be as early as possible (and thus deltas to be as
>> late and "appending" as possible). That could give more readable diffs,
>> especially when combined with heuristics for preferring chunks starting
>> with the lowest amount of indentation.
>>
>> One lesson from these changes seems to be that it is a problem that we
>> use the same low level diff algorithm for revlog delta storage and
>> bundles and for readable patch diffs. One idea that got mentioned at the
>> latest sprint was to use zstandard for storage and "just" seed it with
>> the "a" version of the file as dictionary and let it compress the "b"
>> side. That might be a better long term solution.
>>
>
> That's a nice and funny solution I wonder what are the performance and
> compression level we gains from that.


Using zstd dictionary compression for alternate revlog delta storage was
proposed. However, experimenting with dictionary compression is far down my
priority list for zstd because it requires figuring out how to store
dictionary data. It is much simpler to use zstd as essentially a drop-in
replacement for zlib and figure out the dictionary foo later.

My priorities for zstd are roughly:

1. wire protocol for bundle exchange
2. bundle compression
3. revlog storage
4. dictionary compression and anything else related to changing revlog
"schema"


>
>
> More short term, I wonder how much we could gain from somehow teaching
>> bdiff to consider both parents for each chunk instead of just using
>> deltas from one side and store chunks from the other verbatim. I think
>> that could make a significant difference for repositories with a lot of
>> big merges in files or the manifest.
>>
>> /Mads
>>
>> _______________________________________________
>> Mercurial-devel mailing list
>> Mercurial-devel at mercurial-scm.org
>> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>>
>
> --
> Pierre-Yves David
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20161107/b28fb1a5/attachment.html>