Desired use case for obsmarkers / visibility

[Boris Feld]

On Thu, 2017-11-02 at 10:06 -0700, Gregory Szorc wrote:
> I have a potential use case for obsmarkers / visibility that I want
> to run by people to see if it can be supported.
> 
> Changesets are pushed to the Firefox repo via a landing service. This
> service essentially imports changesets [submitted by the author] and
> rebases them onto the repo head.
> 
> Today, when your changesets are landed, you need to perform "garbage
> collection" on your local repo to remove the old versions of
> changesets. We want landed changesets to disappear after `hg pull`
> picks up the rebased versions.
> 
> This is a pretty straightforward scenario and is supported by
> obsmarkers today. If we enabled the writing of obsolescence markers
> in the landing service, things would essentially "just work."
> 
> Here's where things get a little more complicated.
> 
> When changesets are landed to the Firefox repo today, they are first
> pushed to an "integration" repository. Logically, this can be modeled
> as a single repo divided into public and draft parts. e.g.
> 
> o D (draft) (head)
> o C (draft)
> o B (public)
> o A (public) (root)
> 
> When our CI says a changeset is "good," it is promoted to public.
> e.g.
> 
> o D (draft)
> o C (public) (formerly draft)
> o B (public)
> o A (public) (root)
> 
> Today, when we encounter a "bad" changeset, we perform a backout.
> e.g.
> 
> o D' (draft) (backout of D)
> o D (draft)
> o C (public)
> o B (public)
> o A (public) (root)
> 
> Given our push velocity, it is common to have intermediary changesets
> land before a changeset is identified as "bad." This means there are
> changesets between the initial landings and its backout. e.g.
> 
> o D' (draft) (backout of D)
> o E (draft)
> o D (draft)
> o C (public)
> o B (public)
> o A (public) (root)
> 
> The repo with the backouts is eventually published and the final
> history of the repo is littered with "bad" changesets and backouts.
> This causes all kinds of problems for bisection, annotate, file
> history, etc.
> 
> Instead of performing backouts and leaving the final repo history in
> a sub-optimal state, we want to instead "drop" "bad" changesets
> before they are published. e.g.
> 
> o E' (draft) (rebased from discarded D to C)
> |     x D (draft) (discarded)
> o C (public)
> o B (public)
> o A (public) (root)
> 
> Since we can identify "bad" changesets relatively quickly, this would
> enable us to remove the vast majority of backouts and "bad"
> changesets from the final, published repo history.
> 
> Again, obsolescence as it exists today facilitates this. We can
> perform these drops via `hg histedit` (or similar) and the
> appropriate "prune" obsmarkers are written so the canonical repo has
> the appropriate final history.
> 
> However, the way it works today isn't friendly to end-user workflows.
> 
> If we were to deploy this, the following would happen:
> 
> 1) User creates changeset X and submits for landing.
> 2) Landing service rebases to X' and writes X->X' marker.
> 3) X' turns out to be bad and is dropped. X'->null marker is written
> to convey the prune.
> 4) User pulls and sees X->X'->null and hides X because its most
> recent successor is pruned.
> 5) User is left wondering what happened to X. They possibly forget
> they need to fix and reland X.
> 
> This is bad UX. What we want to happen instead is:
> 
> a) User pulls after X' drop and X is still visible.
> b) Something else happens and some form of X remains
> visible/accessible to user
> 
> The server can't expose X' because everyone would see it. We have 1
> head per repo and don't want to be exposing random "bad" changesets
> to everyone. This seems to rule out the traditional evolve solution
> of "touch" a changeset to revive a prune because I'm not sure how
> we'd send X' to only the user that cares about it. There's also no
> way in obsolescence today to unhide X once it has been obsoleted.
> 
> In the obsmarker world of today, the best solution I can think of is
> "delete obsmarkers on the server." If we discarded the X->X' marker
> (or didn't write it until X' became public), the end-user's original
> changeset X wouldn't be hidden on pull because there is no marker on
> the server referencing X. But this approach feels hacky and is extra
> server-side complexity, which I'd prefer to avoid.

First, `hg strip` get ride of X' and the obsmarkers for you, but that
is a more hacky and traumatic for the repository that you will want 
(especially the caches).

Fortunately there are a couple of other low tech solutions available 
with today implementation:


For your usecase. If people are barely pulling from the integration 
repository, the simplest might be to turn the dropped changeset secret 
(instead of pruning it). That way, they are no longer exchanged (nor
is the associated obsolescence markers between X and X').

This is a simple approach available today. However, that won't help 
people who already pulled from -integration.


To have X' disappear from other people repository while still being
nice to the original author, you can use an extra "-rejected"
repository.

  0) user pushed X⁰, for rebased as X¹ but need to be dropped,
  1) Push X¹ to mozilla-rejected,
  2) 'touch' X¹ into X² (inside mozilla-rejected),
  3) prune X¹ in mozilla-integration,
  4) send an email to the user with the reason for the rejection and
the 
url to pull X' again,

Step (3) means there will be a prune marker to be pulled by everybody 
(from integration). But step (2) ensures there is a successor that the 
original user can use to keep working.

I agree it is not the most elegant solution, but that is easily 
available today.

As an extra (5) steps, -rejected could detect any content-divergent 
version of something that got dropped. The new successors would be the 
next version from the legitimate owner. An obsmarkers can then be 
automatically created from X¹ to that newer version of X³.

> 
> I /think/ the new visibility work proposed by Jun, Durham, and others
> might offer some solutions to this problem. Rather than speculate
> based on my limited knowledge of that proposal, I am hoping someone
> with more knowledge could weigh in more definitively.
> 
> It's worth noting that in our proposed workflow, the "integration"
> changesets that are rewritten exist in a separate repository that
> most people don't pull from. This means we could potentially break
> some "rules" about how obsmarkers work since few would notice. But we
> do pull the "integration" repo into the "stable" repo. So presumably
> obsmarkers would propagate to the "stable" repo and be pulled by
> people, where they could cause problems.
> 
> Is there a solution to this use case? FWIW, I think a solution would
> have use beyond Mozilla's walls: I'm pretty sure a lot of people
> would love for the final history of their repo to be cleaner. It's
> just that today's VCS tools (including Git), don't handle distributed
> history rewriting very gracefully, which discourages people from
> having nice things.


I hope one of the low tech solution offered above solves your
immediate problem to let you move forward with your idea.

Getting something better than the low-tech solution offered above is 
going to be a bit harder. The main issues is that we want to actually 
hide the dropped changeset for all people but some (the author/owner). 
The definition of that "owner" is not something we have really
clarified yet.

If we clarifies this ownership things. There are two different 
approaches we can explore in your case.

Server side solution:

     Variant of the 'secret' approach. The X² changeset is not served
to people unless they are recognized as "owning it" during the
discovery phases.

Client side solution:

     Changeset owned by the local repository, but pruned by a non-
owner don't get hidden until something happens (owner actually obsolete
them or something; The non-owner pruning could be a flag in the
obsmarkers set at pruning time).

Cheers,
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel