Hidden Commits in 4.3

[Durham Goode]

On 4/12/17 10:44 PM, Gregory Szorc wrote:
>
> I just read this entire thread and am trying to wrap my head around
> different sides of the argument. I clearly see where Durham, Ryan, and
> others are coming from with "strip-based operations are bad." I think we
> can all agree on that. I can also see how the new hiding mechanism is
> effectively the same as strip-based operations in terms of behavior, but
> without the poor performance and hacks. So it seems like a generally
> good idea.
>
> But I'm having difficulty comprehending Pierre-Yves's opposition to this
> seemingly good proposal. Pierre-Yves has thought about evolution
> concepts more than anyone, so when he raises an alarm, it is worth
> taking the time to listen and understand his concerns so we don't
> jeopardize the future of evolve. However, despite the vast amount of
> words written, it still isn't clear to me exactly what the main
> objection is. But from seeing "global state" and "monotonic" enough
> times and reading through https://www.mercurial-scm.org/wiki/CEDConcept
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mercurial-2Dscm.org_wiki_CEDConcept&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=nuarHzhP1wi1T9iURRCj1A&m=P4L1wyyuErdKv1jvHswuE4jmAIfQ_TO0FXg9EjEkN_o&s=7Ttw0QZoZoUkJujhNsmSQlKkC2kJ29T1Mtj2_r4jEQg&e=>,
> I think I have an idea. (Pierre-Yves: please confirm or correct what I'm
> about to say.)
>
> I think most of Pierre-Yves's concerns are rooted at a *potential*
> intrinsic incompatibility between a) a generic, non-versioned,
> non-monotonic hiding mechanism (the proposal) and b) what is essentially
> obsolescence markers of today. For example, if the new hiding mechanism
> isn't designed with consideration for evolve, evolve/Mercurial would see
> an instruction to hide a changeset but evolve would be powerless to make
> any kind of reasonable judgement based on the "why" and "how" that
> changeset became hidden because that data just isn't recorded. In other
> words, generic hiding lacks the "global state" and "monotonic"
> properties of the obsolescence store and its exchange, making evolve
> less effective. This *would* undermine the ability to ship a high
> quality product: so Pierre-Yves isn't using hyperbole when he says he
> fears that he could deliver evolve depending on how this hiding feature
> is implemented.

I think your analysis is correct. There are trade offs: A completely 
mutable hidden store would hinder certain types of analysis in the 
future. A append-only-obsmarker based hidden would prevent certain types 
of user interactions in the future (ex: unhiding).

I believe optimizing for the 'certain types of analysis' case is 
optimizing for an edge case of an edge case workflow (mutable history 
conflicts within mutable history collaboration) at the expense of the 
common case workflows (local history mutation, commit revival, commit 
hiding mental model, and simple client/server workflows).

In my understanding, the actual user experience impact on evolve of this 
proposal (vs the theory oriented impact) is that *if* the user is doing 
mutable history collaboration, and *if* they have a mutable history 
conflict of some type, and *if* they've messed with visibility 
themselves, *then* evolve would be unable to determine the correct 
result and would have to prompt the user. I might be over simplifying, 
so I agree that concrete examples would help understand the impact of 
this change.

So, in my head having 'hg hide/unhide' be a dead simple, understandable 
(for both users and extensions) primitive in all cases is highly 
desirable, and is worth the trade off.

> Regardless of whether this is actually Pierre-Yves's fundamental
> concern, there is a conflict here that needs addressed. And I /think/
> that conflict can be summarized as "how do we ensure that evolve is the
> sole source of truth for hidden/obsolescence data?" Because if evolve
> isn't the sole source of truth for hidden/obsolescence data, then evolve
> has to essentially guess at what to do with changesets hidden for
> "unknown" reasons and that severely undermines the ability of evolve to
> work well.

As mentioned above, it only undermines evolve's ability if the user 
actually did manual visibility manipulations. If they didn't, then 
things work exactly as they do today. And the remediation in the failure 
case is to just tell the user "hey, you have to choose between action A 
or action B". This doesn't sound so bad, especially if this prompt is 
going to be part of certain evolve workflows anyway (since there will 
almost certainly exist ambiguous situations in even the most well 
designed evolve world).

> What I think this means is that we need writes to a generic hiding
> mechanism and evolve/obsolescence markers to be mutually exclusive. I.e.
> if obsolescence markers are active, we can't be hiding things via
> not-obsolescence-markers because doing so would deprive evolve of
> critical knowledge.

We'll want obsmarker data even in a non-evolve world. See my restack 
proposal for how obs information can be useful as simply metadata about 
history, regardless of how hidden is computed. But, the two algorithms 
for hiding could be mutually exclusive (with obsmarkers enabled for both).

> So, we can have a generic hiding mechanism in core Mercurial. We can
> lean on that mechanism in the vanilla Mercurial configuration to hide
> changesets so we don't have to strip. And, depending on how it is
> implemented, obsolescence markers could potentially use that mechanism
> for efficient hidden calculation (e.g. if it is a bitmap on disk
> computed from obsolescence markers data). But what is extremely
> undesirable from the perspective of evolve are writes to that hidden
> "store" coming from not-evolve when evolve is active.
>
> If my assessment is correct, I think I have a relatively simple solution
> to this mess:
>
> 1) a unified API in core for "hiding" changesets. It writes directly to
> a generic store or to obsolescence (if enabled). This is the only way to
> "hide" a changeset.
>
> 2) clients refusing to write generic hidden data when obsolescence is
> used. This will prevent evolve from encountering hidden changesets
> lacking "global state" and "monotonic" properties.
>
> 3) any user-facing commands or behavior related to generic hiding (such
> as `hg update <hidden>` unhiding a changeset) need to be disabled or
> changed when obsolescence is enabled because they break the needs of
> evolve to have "global state" and "monotonic" properties. (If #2 is
> implemented, anything attempting to hide/unhide without obsolescence
> markers would break. So this point follows as a consequence of #2.)
>
> 4) a mechanism to "upgrade" a repo using generic hiding data to use
> obsolescence. This may involve some kind of special, local-only
> obsolescence marker that basically says "obsoleted in local storage" so
> when that changeset is pulled from a remote, it can be unhidden without
> the the nasty implications of breaking "global state" and "monotonic"
> properties (because these properties didn't exist in the first place).
> Alternatively, we could just say "sorry, you can't use both at the same
> time: run `hg debugupgraderepo --gc` to prune the store of pre-evolve
> data" and then evolve can "own" all hidden data from what is effectively
> a fresh clone.
>
> #3 poses some interesting UX problems. You ideally want end-user facing
> concepts and commands to remain similar for evolve and non-evolve users.
> But if you want evolve to have its "global state" and "monotonic"
> properties, that necessitates significant behavioral divergence from a
> generic hiding mechanism (e.g. you have to use a new changeset to unhide
> something to satisfy monotonic). We can mitigate the problem by limiting
> the surface area of user-facing commands for hiding/unhiding things.
> And/or we can just say "evolve changes everything" and not worry too
> much about the user impact.

Sure, we could have a flag that toggles you between these two worlds. 
But I think that just punts the deprecation of one down the line and 
leaves extensions in a weird state where they have to be aware of the 
two possible environments (much as they do today with strip-vs-evolve 
environments).