[PATCH 1 of 4 V2] obsolete: track node versions

[Pierre-Yves David]

Summary
-------

I think we need to take an extra step back in the current discussion. We 
seems be get slowly lost in a sub branch of evolution design, discussing 
sub-concerns with a partial view of the big picture.

Currently Evolution has:
* an overall consistent design,
* a planned final UI that allow to introduce complexity progressively,
* a large and diverse base of tester providing useful feedback that 
prompt small adjustment and polishing,
* a defined road-map,
* resource and founding[1] (at least partially) to bring this plan to 
completion.

They are currently no large unsolved problem in this plan that I'm aware 
of and no feedback from testers that would justify and overall full 
redesign of this plan.

If we want to start large discussion impacting evolution we should start 
from the basic. What is unclear about the current plan? What is missing 
from the Roadmap? etc…
We needs to have a common understanding of the situation before digging 
into building solutions.


The proposal at hand disable a corner-stone of evolution (hiding of 
obsolete commit) that has been working well for years. To try solving 
undefined problem and helps with advances-corner-case like 
hash-preservation. This is the wrong kind of trade-off and a direction 
we should not pursue.

I'm made more detailed comments about the actual proposal inline.

[1] Thanks goes to Unity and Bitbucket for that.

On 03/30/2017 08:28 PM, Durham Goode wrote:
> Let's step back a moment and think about what obsmarkers are used for.
> They are used to hide commits, and to automatically perform rebases.

Actually this is a limited view of what evolve aims at, is and currently 
achieves. I'll try to find some time during the freeze to go over the 
existing documentation and clarify some of the core concepts and goal.

> The concerns around obscycles is that allowing cycles (without a perfect
> version scheme) could affect those two uses.  For hiding, it could
> result in hiding commits from the user that they didn't mean to be
> hidden. For rebasing, it means we could rebase onto the wrong successor.

While your conclusion are overall good according your premises, this 
analyses is missing multiples areas where obsmarker are used today (eg: 
troubles detections, checkheads post processing, etc). See my original 
reply to Jun proposal for a list of area to take in account while 
working on evolution related proposal.

> I think the underlying issue is that we're trying to do too much magic
> for the user, and we're unable to find a perfect design to make that
> magic safe and understandable. I think finding the right answer here may
> be impossible.

So far, testing show that automated behaviors of evolution are 
understood by testers and performing their duties (yes, they are some 
known case that needs adjustment). And we have a full plan to deliver 
that to all users.
If you think they are problematic area or unsolved/unsolvable problems 
blocking that goal, can you be more specific and bring them up in a more 
top level discussion about the current evolution plan.

> Proposal
> ===
>
> I propose a series of changes to reduce the magic and remove the need
> for obs versioning, while maintaining the power and increasing the
> understandability of these workflows. It has two parts:
>
>
> 1. Never hide a commit during hg pull. Only hide commits when the user
> does an action (strip/rebase/amend/histedit/evolve)
> ---
>
> One of the recurring issues with obsmarkers is that commits may
> magically disappear when you pull from another repo.

This is not something raised by current testers using full evolution and 
this is not a known weakness of the current design. So I'm not sure why 
you make it a high profile issue.

This comes up often when discussing corner case impacted by proposal or 
as blocker for case were people wants to abuse obsolescence marker out 
of their primary usage. This also comes up in the unorthodox work-flow 
of the Mercurial project, but it is a work-flow issue on the Mercurial side.

> This has two issues: A) we have no good way of explaining it to the user,

This does not match user feedbacks. And it does not seems hard to explain.

    Mercurial hide obsoletes changesets (changeset with a newer version 
(successors) or markers as pruned).
    (unless they have non-obsolete descendants).

 >and B) we can't even be sure the user wants those commits to disappear.

The current set of evolution commands is designed to be intend-full.
Testing has not shown large issue with them. As long as obsolescence 
markers are not used abusively.

> I propose we *never* hide a commit when doing hg pull. When the user
> pulls, we download the obsmarkers like normal, but the commits don't
> disappear. Instead, we show the "[amended|rebased] to HASH" text in a
> log/smartlog output so the user can know the old commits are dead, then
> they can strip them at their leisure. This has the added benefit of
> making it user visible what obsmarker changes the pull brought in.

This removes a important central feature of evolution that has been 
giving good results for the past 5 years.


Let dive in more details on some of the problematic aspect:

* Manually removing obsolete changesets is a tedious and error prone 
process. In addition it will not scale past a small number of changesets 
with very few people involved. I've done this kind of manual stripping 
of old markers in proto-version of evolution and this is not a pleasant 
experience.

* Once you get multiple people involved it will become pretty hard to 
guess other people intention after the fact without reliable data to do 
so. Making it impractical to use for multiple people collaborating with 
evolution. one of the very core use case of evolution.

* Predictable obsmarkers → visibility rules do ensure a consistent 
global state. Drop the rules also loose us the global state.
   Having local actions and situations influencing the visibility means 
different repository with the same global data will display different 
result. With no reliable way to unify they vision. This makes 
communication and collaboration around these states much harder when needed.

As a side note, handling "problematic" action (like cases someone 
wrongly rewrite other peoples changeset) on the "suffering" side is a 
bad pattern as the issue already happens. Pro-actively helping the 
"faulty" side to avoid doing the problematic action in the first place 
is better design. These are the ideas behind phases for example.

> This proposal has two optional side features:
>
> 1a. *Do* hide commits during pull if the pulled version is public.
> 1b. Add an option to strip/prune to auto hide any visible commits that
> have visible successors and don't have visible descendants. So "hg pull
> && hg strip/prune --obsolete-commits" would be roughly equivalent to hg
> pull today.

Having an automated command to apply hiding from obsmarkers might seems 
like a tempting solution to issues regarding tediousness, 
error-proneness and lack of scalability.

However, this is compound as a False solution. In the general cases, 
users are just going to hammer this shinny command over and over, 
getting the same behavior as before but with as extra annoying step. In 
addition giving users will brainlessly apply this automated commands, 
its behavior will needs to be free from surprising corner cases and 
trap, moving it to be equivalent to the same constraint as the current 
behavior of automatically hiding.

> This proposal requires a notable change to core:
>
> - Hidden must be separated from obsmarkers storage and mutable outside
> obsmarkers. This is possible with Jun's proposed phaseroot-esque hidden
> storage.
>
>
> 2. Auto rebase uses "visible successors" instead of "latest successor"
> ---
>
> Today auto rebase (i.e. evolve) uses the latest successor as the
> destination of the rebases. I propose we change that to "visible
> successor" instead, where visible successor is defined as "any successor
> commit that is not hidden". This means, regardless of the current
> obsmarkers setup, the destination of the auto rebase is whatever
> successor is currently visible. Which is probably what the user wanted
> anyway.

(note: currently latest successors == visible successors, so behavior is 
close to what is described here)

> If there are multiple visible successors, auto rebase fails and shows a
> list of the potential visible successors.

(note: currently evolve will error out if they are ambigious destination 
successors (divergence, some split case), so the behavior proposal is close)

> Each item in the list can have  the "[amended|rebased] to HASH" string next to it so users can
> understand at a glance the ordering and make a decision.

I'm curious about how this proposal handles the cases when split is 
involved. (not saying it cannot handled them well, just want more details).

> They can either
> use -d on rebase to specify a solution to the conflict, or they  can use
> the normal visibility commands (hg strip/prune) to remove any
> undesirable commits.

To clarify. `hg prune` is not a visibility commands. It is a command 
used to globally discard an old/unwanted changesets.
This is an important distinction to understand for one who want to work 
around evolution design.

> The presence of cycles does not affect this at all, and there is no need
> for marker versioning. Auto rebase still uses whatever successors are
> visible, even if the successor is "older" than it, because of a cycle.
> The user can use the same rebase -d or strip/prune resolutions to
> resolve the conflict.
>
> Summary of what these two changes achieve
> ===
>
> From a single, non-exchanging user's perspective the above proposal does
> not affect current UX around when things are hidden (like during
> rebase/amend/etc), but does allows all the benefits of the obscycle
> discussion (allowing unhiding) without the need for obsmarker versioning.

Note: the founding goal of evolution is to allow people to exchange and 
collaborate on mutable history in a distributed manners. If new ideas or 
proposals puts the local only usecase first it is a hint they probably 
leave important area related to distributed case out. As this proposal 
do (eg: behavior on push?, checkheads, handling of prune, etc…)


> From an exchange user's perspective, this makes exchange much more
> deterministic for the user (nothing is magically removed from the repo,
> and what new obs information from the pull is explained via smartlog),

Current hiding behavior is already deterministic and allow clear rules 
(nothings magical). Testers are requesting access to more details about 
what happens during a push to help in some case. Such improvement is 
founded and scheduled so it will happen soon. We'll push and emphasis on 
changes affecting the user changesets for further clarity.

> while still enabling auto rebase workflows. It also makes obsmarker
> conflict (divergence/etc) easier to understand and resolve by allowing
> users to resolve obsmarker conflicts using tools they're already
> familiar with (rebase -d, strip/prune).

I do not understand what aspect of obsmarkers conflict you are currently 
flagging as "hard to understand" neither how your proposal makes their 
resolution simpler. Can you elaborate?

Cheers,

-- 
Pierre-Yves David