[PATCH 1 of 3] bundle: allow creation of changegroup3 bundles

Tue May 9 02:18:33 EDT 2017

On 05/08/2017 09:34 PM, Gregory Szorc wrote:
>
>
>> On May 6, 2017, at 02:50, Pierre-Yves David <pierre-yves.david at ens-lyon.org> wrote:
>>
>>
>>
>>> On 05/05/2017 12:14 AM, Gregory Szorc wrote:
>>> On Thu, May 4, 2017 at 12:14 PM, Pierre-Yves David
>>> <pierre-yves.david at ens-lyon.org <mailto:pierre-yves.david at ens-lyon.org>>
>>> wrote:
>>>
>>>
>>>
>>>    On 05/04/2017 07:04 PM, Martin von Zweigbergk via Mercurial-devel wrote:
>>>
>>>        # HG changeset patch
>>>        # User Martin von Zweigbergk <martinvonz at google.com
>>>        <mailto:martinvonz at google.com>>
>>>        # Date 1458952398 25200
>>>        #      Fri Mar 25 17:33:18 2016 -0700
>>>        # Node ID f4b255ff67d05e329cc3d80ce1552f30005f58e9
>>>        # Parent  06058cbb9209df54bfa0aaf9f7d10b836a1e3ee9
>>>        bundle: allow creation of changegroup3 bundles
>>>
>>>        With this change, one can use "hg bundle -t v3" to creat
>>>        changegroup3
>>>        bundles, which support treemanifests and revlog flags. In
>>>        treemanifest
>>>        repos, v3 is the default bundle format.
>>>
>>>
>>>    The bundle spec "version" cover a collection of setting for the
>>>    underlying content. It is not directly correlated to the changegroup
>>>    version.
>>>
>>>    Tree manifest is still experimental so it seems a bit early to burn
>>>    a new bundle spec version on this especially since we have other
>>>    bundle improvement coming (phase, tag cache, obsmarkers, etc).
>>>
>>>    So I don't think we should introduce a new bundlespec version yet.
>>>    Instead I would recommend the following:
>>>
>>>    1) introduce a parameter to control de cg-version (cgversion=03?)
>>>    2) automatically bump the cg version to '03' if no spec is passed
>>>    and tree manifest is used,
>>>    3) late in the 4.3 cycles consider introducing a v3 with all the new
>>>    improvements (if some make sense at the user-visible feature level).
>>>
>>>
>>> There was confusion about this in IRC.
>>>
>>> I agree with what Pierre-Yves said: while this will require a "v3" to
>>> support cg3, it is still a bit too early in cg3's support cycle to
>>> introduce a stable, user-facing bundle spec version. Until we have a
>>> better API for controlling bundle generation via bundle specs, I think
>>> this should be hacked in via an experimental config option while
>>> recycling "v2." Alternatively, we can name the version "expcg3" or
>>> something less stable and rename it (presumably to "v3") once the
>>> feature is stable. The latter will make it much easier to test the
>>> feature and won't result in accidentally producing "v2" bundles that
>>> legacy clients choke on.
>>>
>>> The concern with the "vN" versions is that every Mercurial client with
>>> knowledge of that version string has to be capable of handling any
>>> future bundle generated with that same "vN" version (or have code for
>>> filtering from other data in the clone bundles manifest, preferably from
>>> BUNDLESPEC). This means every time there is a new required bundle part,
>>> we have to invent a new "vN" or find some other way to alter
>>> clonebundles entries so old clients know they can't consume them.
>>
>> TL;DR; We needs the flexibility of params based spec. We should start there and care about the "vN" version later in the cycle.
>>
>> About bundle parameters
>> -----------------------
>>
>> (putting compatibility question aside for now)
>>
>> There are two dimensions in play here:
>>
>> 1) data format (how we encode the information),
>> 2) data content (what data do we encore in the bundle).
>>
>> Formats makes it "easy" to think in terms of linear increasing version number, but it is less clear for content. The mix of both makes it more complex.
>>
>> The above make me think we should aim for a parameter centric solution here.
>>
>> There are multiple kinds of data one -may- want includes into a bundles.
>>
>> * changesets,
>> * phases,
>> * bookmarks,
>> * obsolescence,
>> * (and various caches)
>>
>> I do not think there will be any definitive answer here. For example people might want to generate a bundle with or without bookmarks. So we needs a reasonable "interface" for users to select the bundle content. bundlespec parameters seems like the way to go. (eg: "v3;phases=1;bookmarks=0;obsolescence=0")
>>
>> In addition to that, there are the question of -how- these data are encoded (eg: changegroup version). Since there are multiple types of data, it seems useful to be able to control version used for each part (eg: v3;cg.version=04;obsolescence.version=05).
>>
>>
>> Of course, we should have a reasonable interface for our simple user.
>> So, the way I see the bundlespec's "vN" is more collection of default for the various value described above. picking a "v3" will select value for the known parameters, but parameters will override them. For example:
>>
>> * v2 → cg.version=02;phases=no;bookmarks=no;obsolescence=no
>> * v3 → cg.version=03;phases=yes;bookmarks=no;obs…=yes;obs….version=02
>>
>> And the following would be valid
>>
>> * v3;bookmark=yes
>> * v3;cg.version=02
>>
>> Depending of the features in use in a repository, a default version is picked:
>>
>> For example:
>> * old format -> v1
>> * general delta -> default v2+
>> * tree manifest -> default v3+
>>
>> About compatibility
>> -------------------
>>
>> Gregory is raising a valid concern regarding bundle parameters. Right now they are ignored by the code and all existing (clonebundle) client do not process them at all.
>>
>> However, this concerns is clonebundle specific. As long as we do not exposes `v2;pa=rams` bundles in clonebundle manifest we are in the clear. other usages will be fine with using parameters with v2.
>>
>> In all cases, old client will always mis-behave when it comes to the behavior of `hg bundle v2;any=params;values=foo;` I think we'll have to lives with that :-/.
>>
>> Way forward
>> -----------
>>
>> Here is a proposal of how to move forward on this topic
>>
>> 1) focus on defining the parameters first,
>>
>> 2) add safety to ensure we do not generate clone bundle manifest using "v2;pa=rams"
>>
>> 3) decide on the value of "v3" around the end of the 4.3 cycle.
>>
>> What do you think?
>
> This all sounds pretty reasonable and was mostly along the lines of what I was thinking when I introduced a more formal API for bundlespecs.
>
> FWIW the use of vN isn't strictly necessary: a bundlespec is really a collection of features. The vN is useful to concisely express support for a well-defined set of features and/or defaults. Note that producers (hg bundle) mainly care about defaults and consumers (clonebundles) care about features/compatibility. Something we may want to do is have v3 map to a set of defaults and to always encode those params in the bundlespec when serialized. So "v3" would only be accepted on user-facing commands but clonebundles would always use "v3;changegroup=02:bookmarks=01" etc.

If I understand your idea correctly, you proposed focucing vN for input 
(hg bundle --type vN) and having as explicit as possible spec in 
clonebundle manifest. That seems reasonable.

Martin, do you have a clear enough view of how to move forward with a 
bundle parameters or do you need extra guidance ?

Cheers,

-- 
Pierre-Yves David