[PATCH 1 of 4 V2] obsolete: track node versions

[Ryan McElroy]

On 4/5/17 11:23 AM, Pierre-Yves David wrote:
> Summary
> -------
>
> I think we need to take an extra step back in the current discussion. 
> We seems be get slowly lost in a sub branch of evolution design, 
> discussing sub-concerns with a partial view of the big picture.

I agree. Overall, my feeling is that trying to redesign changeset 
evolution is not what we should be concentrating on.

>
> Currently Evolution has:
> * an overall consistent design,
> * a planned final UI that allow to introduce complexity progressively,
> * a large and diverse base of tester providing useful feedback that 
> prompt small adjustment and polishing,
> * a defined road-map,
> * resource and founding[1] (at least partially) to bring this plan to 
> completion.

s/founding/funding :-p

>
> They are currently no large unsolved problem in this plan that I'm 
> aware of and no feedback from testers that would justify and overall 
> full redesign of this plan.
>
> If we want to start large discussion impacting evolution we should 
> start from the basic. What is unclear about the current plan? What is 
> missing from the Roadmap? etc…
> We needs to have a common understanding of the situation before 
> digging into building solutions.
>
>
> The proposal at hand disable a corner-stone of evolution (hiding of 
> obsolete commit) that has been working well for years. To try solving 
> undefined problem and helps with advances-corner-case like 
> hash-preservation. This is the wrong kind of trade-off and a direction 
> we should not pursue.
>
> I'm made more detailed comments about the actual proposal inline.
>
> [1] Thanks goes to Unity and Bitbucket for that.
>
> On 03/30/2017 08:28 PM, Durham Goode wrote:
>> Let's step back a moment and think about what obsmarkers are used for.
>> They are used to hide commits, and to automatically perform rebases.
>
> Actually this is a limited view of what evolve aims at, is and 
> currently achieves. I'll try to find some time during the freeze to go 
> over the existing documentation and clarify some of the core concepts 
> and goal.

I agree that changeset evolution is a bigger concept. A lot of these 
proposals seem to miss that exchange is the hard part that evolution has 
concentrated on and gotten remarkably good at (I've actually used 
full-on evolve workflows in collaboration with Sean Farley while working 
on remote names, and it *actually works* when editing the same series of 
changesets! We probably shouldn't break all of this).

The problems FB is trying to solve are actually less ambitious right 
now: we want to solve local-user workflows. We feel that changeset 
evolution has made some difficult choices in this area (like not 
reviving old hashes) that make single-user workflows harder to reason 
about, but these are choices it had to make to get the more ambitious 
plan to work well (granted, I don't claim to fully understand all the 
implications of obsmarker cycles -- I haven't had the cycles [heh] to 
deeply understand that yet).

I think instead of fighting about the future of changeset evolution, we 
can concentrate on building the common infrastructure for hiding commits 
that we all want and need to be fast and scalable. Then we can build 
whatever experiences we want on top of that, using obsolescence, 
archiving, internal-phase, or whatever other concepts we come up with. 
We should be able to agree on an underlying system to do this in a 
performant and scalable way that lets other extensions do more 
experimental things with what gets hidden or unhidden.

>
>> The concerns around obscycles is that allowing cycles (without a perfect
>> version scheme) could affect those two uses.  For hiding, it could
>> result in hiding commits from the user that they didn't mean to be
>> hidden. For rebasing, it means we could rebase onto the wrong successor.
>
> While your conclusion are overall good according your premises, this 
> analyses is missing multiples areas where obsmarker are used today 
> (eg: troubles detections, checkheads post processing, etc). See my 
> original reply to Jun proposal for a list of area to take in account 
> while working on evolution related proposal.
>
>> I think the underlying issue is that we're trying to do too much magic
>> for the user, and we're unable to find a perfect design to make that
>> magic safe and understandable. I think finding the right answer here may
>> be impossible.
>
> So far, testing show that automated behaviors of evolution are 
> understood by testers and performing their duties (yes, they are some 
> known case that needs adjustment). And we have a full plan to deliver 
> that to all users.
> If you think they are problematic area or unsolved/unsolvable problems 
> blocking that goal, can you be more specific and bring them up in a 
> more top level discussion about the current evolution plan.
>
>> Proposal
>> ===
>>
>> I propose a series of changes to reduce the magic and remove the need
>> for obs versioning, while maintaining the power and increasing the
>> understandability of these workflows. It has two parts:
>>
>>
>> 1. Never hide a commit during hg pull. Only hide commits when the user
>> does an action (strip/rebase/amend/histedit/evolve)
>> ---
>>
>> One of the recurring issues with obsmarkers is that commits may
>> magically disappear when you pull from another repo.

I disagree that this is a problem. Most of the user reports I'm aware of 
are more concerned with why things we "know are old" need to be cleaned 
up manually at all. User experience can also be improved here, for 
example via more explicitly telling the user what changed, but I don't 
think "things becoming obsolete and possibly hidden on pull" is a 
confusing user experience (see our own pullcreatemarkers extension as an 
example).

>
> This is not something raised by current testers using full evolution 
> and this is not a known weakness of the current design. So I'm not 
> sure why you make it a high profile issue.
>
> This comes up often when discussing corner case impacted by proposal 
> or as blocker for case were people wants to abuse obsolescence marker 
> out of their primary usage. This also comes up in the unorthodox 
> work-flow of the Mercurial project, but it is a work-flow issue on the 
> Mercurial side.
>
>> This has two issues: A) we have no good way of explaining it to the 
>> user,
>
> This does not match user feedbacks. And it does not seems hard to 
> explain.
>
>    Mercurial hide obsoletes changesets (changeset with a newer version 
> (successors) or markers as pruned).
>    (unless they have non-obsolete descendants).
>
> >and B) we can't even be sure the user wants those commits to disappear.
>
> The current set of evolution commands is designed to be intend-full.
> Testing has not shown large issue with them. As long as obsolescence 
> markers are not used abusively.
>
>> I propose we *never* hide a commit when doing hg pull. When the user
>> pulls, we download the obsmarkers like normal, but the commits don't
>> disappear. Instead, we show the "[amended|rebased] to HASH" text in a
>> log/smartlog output so the user can know the old commits are dead, then
>> they can strip them at their leisure. This has the added benefit of
>> making it user visible what obsmarker changes the pull brought in.
>
> This removes a important central feature of evolution that has been 
> giving good results for the past 5 years.
>
>
> Let dive in more details on some of the problematic aspect:
>
> * Manually removing obsolete changesets is a tedious and error prone 
> process. In addition it will not scale past a small number of 
> changesets with very few people involved. I've done this kind of 
> manual stripping of old markers in proto-version of evolution and this 
> is not a pleasant experience.
>
> * Once you get multiple people involved it will become pretty hard to 
> guess other people intention after the fact without reliable data to 
> do so. Making it impractical to use for multiple people collaborating 
> with evolution. one of the very core use case of evolution.
>
> * Predictable obsmarkers → visibility rules do ensure a consistent 
> global state. Drop the rules also loose us the global state.
>   Having local actions and situations influencing the visibility means 
> different repository with the same global data will display different 
> result. With no reliable way to unify they vision. This makes 
> communication and collaboration around these states much harder when 
> needed.
>
> As a side note, handling "problematic" action (like cases someone 
> wrongly rewrite other peoples changeset) on the "suffering" side is a 
> bad pattern as the issue already happens. Pro-actively helping the 
> "faulty" side to avoid doing the problematic action in the first place 
> is better design. These are the ideas behind phases for example.
>
>> This proposal has two optional side features:
>>
>> 1a. *Do* hide commits during pull if the pulled version is public.
>> 1b. Add an option to strip/prune to auto hide any visible commits that
>> have visible successors and don't have visible descendants. So "hg pull
>> && hg strip/prune --obsolete-commits" would be roughly equivalent to hg
>> pull today.
>
> Having an automated command to apply hiding from obsmarkers might 
> seems like a tempting solution to issues regarding tediousness, 
> error-proneness and lack of scalability.
>
> However, this is compound as a False solution. In the general cases, 
> users are just going to hammer this shinny command over and over, 
> getting the same behavior as before but with as extra annoying step. 
> In addition giving users will brainlessly apply this automated 
> commands, its behavior will needs to be free from surprising corner 
> cases and trap, moving it to be equivalent to the same constraint as 
> the current behavior of automatically hiding.
>
>> This proposal requires a notable change to core:
>>
>> - Hidden must be separated from obsmarkers storage and mutable outside
>> obsmarkers. This is possible with Jun's proposed phaseroot-esque hidden
>> storage.
>>
>>
>> 2. Auto rebase uses "visible successors" instead of "latest successor"
>> ---
>>
>> Today auto rebase (i.e. evolve) uses the latest successor as the
>> destination of the rebases. I propose we change that to "visible
>> successor" instead, where visible successor is defined as "any successor
>> commit that is not hidden". This means, regardless of the current
>> obsmarkers setup, the destination of the auto rebase is whatever
>> successor is currently visible. Which is probably what the user wanted
>> anyway.
>
> (note: currently latest successors == visible successors, so behavior 
> is close to what is described here)
>
>> If there are multiple visible successors, auto rebase fails and shows a
>> list of the potential visible successors.
>
> (note: currently evolve will error out if they are ambigious 
> destination successors (divergence, some split case), so the behavior 
> proposal is close)
>
>> Each item in the list can have  the "[amended|rebased] to HASH" 
>> string next to it so users can
>> understand at a glance the ordering and make a decision.
>
> I'm curious about how this proposal handles the cases when split is 
> involved. (not saying it cannot handled them well, just want more 
> details).
>
>> They can either
>> use -d on rebase to specify a solution to the conflict, or they can use
>> the normal visibility commands (hg strip/prune) to remove any
>> undesirable commits.
>
> To clarify. `hg prune` is not a visibility commands. It is a command 
> used to globally discard an old/unwanted changesets.
> This is an important distinction to understand for one who want to 
> work around evolution design.
>
>> The presence of cycles does not affect this at all, and there is no need
>> for marker versioning. Auto rebase still uses whatever successors are
>> visible, even if the successor is "older" than it, because of a cycle.
>> The user can use the same rebase -d or strip/prune resolutions to
>> resolve the conflict.
>>
>> Summary of what these two changes achieve
>> ===
>>
>> From a single, non-exchanging user's perspective the above proposal does
>> not affect current UX around when things are hidden (like during
>> rebase/amend/etc), but does allows all the benefits of the obscycle
>> discussion (allowing unhiding) without the need for obsmarker 
>> versioning.
>
> Note: the founding goal of evolution is to allow people to exchange 
> and collaborate on mutable history in a distributed manners. If new 
> ideas or proposals puts the local only usecase first it is a hint they 
> probably leave important area related to distributed case out. As this 
> proposal do (eg: behavior on push?, checkheads, handling of prune, etc…)
>
>
>> From an exchange user's perspective, this makes exchange much more
>> deterministic for the user (nothing is magically removed from the repo,
>> and what new obs information from the pull is explained via smartlog),
>
> Current hiding behavior is already deterministic and allow clear rules 
> (nothings magical). Testers are requesting access to more details 
> about what happens during a push to help in some case. Such 
> improvement is founded and scheduled so it will happen soon. We'll 
> push and emphasis on changes affecting the user changesets for further 
> clarity.
>
>> while still enabling auto rebase workflows. It also makes obsmarker
>> conflict (divergence/etc) easier to understand and resolve by allowing
>> users to resolve obsmarker conflicts using tools they're already
>> familiar with (rebase -d, strip/prune).
>
> I do not understand what aspect of obsmarkers conflict you are 
> currently flagging as "hard to understand" neither how your proposal 
> makes their resolution simpler. Can you elaborate?
>
>

I strongly believe that going back into a long discussion or battle over 
how evolve should behave is a good use of anyone's time. Can't we just 
build out core hiding mechanisms and build experiences on top of that? 
We can each use this in the way we feel best.

Okay, I'm really going to read Durham's proposal now (things keep coming 
up, even on "no meeting wednesday :-p)