Desired use case for obsmarkers / visibility

Boris Feld boris.feld at octobus.net
Mon Nov 13 11:33:15 EST 2017


On Fri, 2017-11-03 at 20:34 +0100, Boris Feld wrote:
> On Thu, 2017-11-02 at 10:06 -0700, Gregory Szorc wrote:
> > I have a potential use case for obsmarkers / visibility that I want
> > to run by people to see if it can be supported.
> > 
> > Changesets are pushed to the Firefox repo via a landing service.
> > This
> > service essentially imports changesets [submitted by the author]
> > and
> > rebases them onto the repo head.
> > 
> > Today, when your changesets are landed, you need to perform
> > "garbage
> > collection" on your local repo to remove the old versions of
> > changesets. We want landed changesets to disappear after `hg pull`
> > picks up the rebased versions.
> > 
> > This is a pretty straightforward scenario and is supported by
> > obsmarkers today. If we enabled the writing of obsolescence markers
> > in the landing service, things would essentially "just work."
> > 
> > Here's where things get a little more complicated.
> > 
> > When changesets are landed to the Firefox repo today, they are
> > first
> > pushed to an "integration" repository. Logically, this can be
> > modeled
> > as a single repo divided into public and draft parts. e.g.
> > 
> > o D (draft) (head)
> > o C (draft)
> > o B (public)
> > o A (public) (root)
> > 
> > When our CI says a changeset is "good," it is promoted to public.
> > e.g.
> > 
> > o D (draft)
> > o C (public) (formerly draft)
> > o B (public)
> > o A (public) (root)
> > 
> > Today, when we encounter a "bad" changeset, we perform a backout.
> > e.g.
> > 
> > o D' (draft) (backout of D)
> > o D (draft)
> > o C (public)
> > o B (public)
> > o A (public) (root)
> > 
> > Given our push velocity, it is common to have intermediary
> > changesets
> > land before a changeset is identified as "bad." This means there
> > are
> > changesets between the initial landings and its backout. e.g.
> > 
> > o D' (draft) (backout of D)
> > o E (draft)
> > o D (draft)
> > o C (public)
> > o B (public)
> > o A (public) (root)
> > 
> > The repo with the backouts is eventually published and the final
> > history of the repo is littered with "bad" changesets and backouts.
> > This causes all kinds of problems for bisection, annotate, file
> > history, etc.
> > 
> > Instead of performing backouts and leaving the final repo history
> > in
> > a sub-optimal state, we want to instead "drop" "bad" changesets
> > before they are published. e.g.
> > 
> > o E' (draft) (rebased from discarded D to C)
> > >      x D (draft) (discarded)
> > 
> > o C (public)
> > o B (public)
> > o A (public) (root)
> > 
> > Since we can identify "bad" changesets relatively quickly, this
> > would
> > enable us to remove the vast majority of backouts and "bad"
> > changesets from the final, published repo history.
> > 
> > Again, obsolescence as it exists today facilitates this. We can
> > perform these drops via `hg histedit` (or similar) and the
> > appropriate "prune" obsmarkers are written so the canonical repo
> > has
> > the appropriate final history.
> > 
> > However, the way it works today isn't friendly to end-user
> > workflows.
> > 
> > If we were to deploy this, the following would happen:
> > 
> > 1) User creates changeset X and submits for landing.
> > 2) Landing service rebases to X' and writes X->X' marker.
> > 3) X' turns out to be bad and is dropped. X'->null marker is
> > written
> > to convey the prune.
> > 4) User pulls and sees X->X'->null and hides X because its most
> > recent successor is pruned.
> > 5) User is left wondering what happened to X. They possibly forget
> > they need to fix and reland X.
> > 
> > This is bad UX. What we want to happen instead is:
> > 
> > a) User pulls after X' drop and X is still visible.
> > b) Something else happens and some form of X remains
> > visible/accessible to user
> > 
> > The server can't expose X' because everyone would see it. We have 1
> > head per repo and don't want to be exposing random "bad" changesets
> > to everyone. This seems to rule out the traditional evolve solution
> > of "touch" a changeset to revive a prune because I'm not sure how
> > we'd send X' to only the user that cares about it. There's also no
> > way in obsolescence today to unhide X once it has been obsoleted.
> > 
> > In the obsmarker world of today, the best solution I can think of
> > is
> > "delete obsmarkers on the server." If we discarded the X->X' marker
> > (or didn't write it until X' became public), the end-user's
> > original
> > changeset X wouldn't be hidden on pull because there is no marker
> > on
> > the server referencing X. But this approach feels hacky and is
> > extra
> > server-side complexity, which I'd prefer to avoid.
> 
> First, `hg strip` get ride of X' and the obsmarkers for you, but that
> is a more hacky and traumatic for the repository that you will want 
> (especially the caches).
> 
> Fortunately there are a couple of other low tech solutions available 
> with today implementation:
> 
> 
> For your usecase. If people are barely pulling from the integration 
> repository, the simplest might be to turn the dropped changeset
> secret 
> (instead of pruning it). That way, they are no longer exchanged (nor
> is the associated obsolescence markers between X and X').
> 
> This is a simple approach available today. However, that won't help 
> people who already pulled from -integration.
> 
> 
> To have X' disappear from other people repository while still being
> nice to the original author, you can use an extra "-rejected"
> repository.
> 
>   0) user pushed X⁰, for rebased as X¹ but need to be dropped,
>   1) Push X¹ to mozilla-rejected,
>   2) 'touch' X¹ into X² (inside mozilla-rejected),
>   3) prune X¹ in mozilla-integration,
>   4) send an email to the user with the reason for the rejection and
> the 
> url to pull X' again,
> 
> Step (3) means there will be a prune marker to be pulled by
> everybody 
> (from integration). But step (2) ensures there is a successor that
> the 
> original user can use to keep working.
> 
> I agree it is not the most elegant solution, but that is easily 
> available today.
> 
> As an extra (5) steps, -rejected could detect any content-divergent 
> version of something that got dropped. The new successors would be
> the 
> next version from the legitimate owner. An obsmarkers can then be 
> automatically created from X¹ to that newer version of X³.
> 
> > 
> > I /think/ the new visibility work proposed by Jun, Durham, and
> > others
> > might offer some solutions to this problem. Rather than speculate
> > based on my limited knowledge of that proposal, I am hoping someone
> > with more knowledge could weigh in more definitively.
> > 
> > It's worth noting that in our proposed workflow, the "integration"
> > changesets that are rewritten exist in a separate repository that
> > most people don't pull from. This means we could potentially break
> > some "rules" about how obsmarkers work since few would notice. But
> > we
> > do pull the "integration" repo into the "stable" repo. So
> > presumably
> > obsmarkers would propagate to the "stable" repo and be pulled by
> > people, where they could cause problems.
> > 
> > Is there a solution to this use case? FWIW, I think a solution
> > would
> > have use beyond Mozilla's walls: I'm pretty sure a lot of people
> > would love for the final history of their repo to be cleaner. It's
> > just that today's VCS tools (including Git), don't handle
> > distributed
> > history rewriting very gracefully, which discourages people from
> > having nice things.
> 
> 
> I hope one of the low tech solution offered above solves your
> immediate problem to let you move forward with your idea.
> 
> Getting something better than the low-tech solution offered above is 
> going to be a bit harder. The main issues is that we want to
> actually 
> hide the dropped changeset for all people but some (the
> author/owner). 
> The definition of that "owner" is not something we have really
> clarified yet.
> 
> If we clarifies this ownership things. There are two different 
> approaches we can explore in your case.
> 
> Server side solution:
> 
>      Variant of the 'secret' approach. The X² changeset is not served
> to people unless they are recognized as "owning it" during the
> discovery phases.
> 
> Client side solution:
> 
>      Changeset owned by the local repository, but pruned by a non-
> owner don't get hidden until something happens (owner actually
> obsolete
> them or something; The non-owner pruning could be a flag in the
> obsmarkers set at pruning time).

By thinking more on the problem, I have a question, does people
actually needs to pull from the integration repo ? If not, a secret-
based workflow might do the job.

Cheers,
Boris


More information about the Mercurial-devel mailing list