Desired use case for obsmarkers / visibility

Gregory Szorc gregory.szorc at gmail.com
Mon Nov 13 11:44:51 EST 2017


On Mon, Nov 13, 2017 at 8:33 AM, Boris Feld <boris.feld at octobus.net> wrote:

> On Fri, 2017-11-03 at 20:34 +0100, Boris Feld wrote:
> > On Thu, 2017-11-02 at 10:06 -0700, Gregory Szorc wrote:
> > > I have a potential use case for obsmarkers / visibility that I want
> > > to run by people to see if it can be supported.
> > >
> > > Changesets are pushed to the Firefox repo via a landing service.
> > > This
> > > service essentially imports changesets [submitted by the author]
> > > and
> > > rebases them onto the repo head.
> > >
> > > Today, when your changesets are landed, you need to perform
> > > "garbage
> > > collection" on your local repo to remove the old versions of
> > > changesets. We want landed changesets to disappear after `hg pull`
> > > picks up the rebased versions.
> > >
> > > This is a pretty straightforward scenario and is supported by
> > > obsmarkers today. If we enabled the writing of obsolescence markers
> > > in the landing service, things would essentially "just work."
> > >
> > > Here's where things get a little more complicated.
> > >
> > > When changesets are landed to the Firefox repo today, they are
> > > first
> > > pushed to an "integration" repository. Logically, this can be
> > > modeled
> > > as a single repo divided into public and draft parts. e.g.
> > >
> > > o D (draft) (head)
> > > o C (draft)
> > > o B (public)
> > > o A (public) (root)
> > >
> > > When our CI says a changeset is "good," it is promoted to public.
> > > e.g.
> > >
> > > o D (draft)
> > > o C (public) (formerly draft)
> > > o B (public)
> > > o A (public) (root)
> > >
> > > Today, when we encounter a "bad" changeset, we perform a backout.
> > > e.g.
> > >
> > > o D' (draft) (backout of D)
> > > o D (draft)
> > > o C (public)
> > > o B (public)
> > > o A (public) (root)
> > >
> > > Given our push velocity, it is common to have intermediary
> > > changesets
> > > land before a changeset is identified as "bad." This means there
> > > are
> > > changesets between the initial landings and its backout. e.g.
> > >
> > > o D' (draft) (backout of D)
> > > o E (draft)
> > > o D (draft)
> > > o C (public)
> > > o B (public)
> > > o A (public) (root)
> > >
> > > The repo with the backouts is eventually published and the final
> > > history of the repo is littered with "bad" changesets and backouts.
> > > This causes all kinds of problems for bisection, annotate, file
> > > history, etc.
> > >
> > > Instead of performing backouts and leaving the final repo history
> > > in
> > > a sub-optimal state, we want to instead "drop" "bad" changesets
> > > before they are published. e.g.
> > >
> > > o E' (draft) (rebased from discarded D to C)
> > > >      x D (draft) (discarded)
> > >
> > > o C (public)
> > > o B (public)
> > > o A (public) (root)
> > >
> > > Since we can identify "bad" changesets relatively quickly, this
> > > would
> > > enable us to remove the vast majority of backouts and "bad"
> > > changesets from the final, published repo history.
> > >
> > > Again, obsolescence as it exists today facilitates this. We can
> > > perform these drops via `hg histedit` (or similar) and the
> > > appropriate "prune" obsmarkers are written so the canonical repo
> > > has
> > > the appropriate final history.
> > >
> > > However, the way it works today isn't friendly to end-user
> > > workflows.
> > >
> > > If we were to deploy this, the following would happen:
> > >
> > > 1) User creates changeset X and submits for landing.
> > > 2) Landing service rebases to X' and writes X->X' marker.
> > > 3) X' turns out to be bad and is dropped. X'->null marker is
> > > written
> > > to convey the prune.
> > > 4) User pulls and sees X->X'->null and hides X because its most
> > > recent successor is pruned.
> > > 5) User is left wondering what happened to X. They possibly forget
> > > they need to fix and reland X.
> > >
> > > This is bad UX. What we want to happen instead is:
> > >
> > > a) User pulls after X' drop and X is still visible.
> > > b) Something else happens and some form of X remains
> > > visible/accessible to user
> > >
> > > The server can't expose X' because everyone would see it. We have 1
> > > head per repo and don't want to be exposing random "bad" changesets
> > > to everyone. This seems to rule out the traditional evolve solution
> > > of "touch" a changeset to revive a prune because I'm not sure how
> > > we'd send X' to only the user that cares about it. There's also no
> > > way in obsolescence today to unhide X once it has been obsoleted.
> > >
> > > In the obsmarker world of today, the best solution I can think of
> > > is
> > > "delete obsmarkers on the server." If we discarded the X->X' marker
> > > (or didn't write it until X' became public), the end-user's
> > > original
> > > changeset X wouldn't be hidden on pull because there is no marker
> > > on
> > > the server referencing X. But this approach feels hacky and is
> > > extra
> > > server-side complexity, which I'd prefer to avoid.
> >
> > First, `hg strip` get ride of X' and the obsmarkers for you, but that
> > is a more hacky and traumatic for the repository that you will want
> > (especially the caches).
> >
> > Fortunately there are a couple of other low tech solutions available
> > with today implementation:
> >
> >
> > For your usecase. If people are barely pulling from the integration
> > repository, the simplest might be to turn the dropped changeset
> > secret
> > (instead of pruning it). That way, they are no longer exchanged (nor
> > is the associated obsolescence markers between X and X').
> >
> > This is a simple approach available today. However, that won't help
> > people who already pulled from -integration.
> >
> >
> > To have X' disappear from other people repository while still being
> > nice to the original author, you can use an extra "-rejected"
> > repository.
> >
> >   0) user pushed X⁰, for rebased as X¹ but need to be dropped,
> >   1) Push X¹ to mozilla-rejected,
> >   2) 'touch' X¹ into X² (inside mozilla-rejected),
> >   3) prune X¹ in mozilla-integration,
> >   4) send an email to the user with the reason for the rejection and
> > the
> > url to pull X' again,
> >
> > Step (3) means there will be a prune marker to be pulled by
> > everybody
> > (from integration). But step (2) ensures there is a successor that
> > the
> > original user can use to keep working.
> >
> > I agree it is not the most elegant solution, but that is easily
> > available today.
> >
> > As an extra (5) steps, -rejected could detect any content-divergent
> > version of something that got dropped. The new successors would be
> > the
> > next version from the legitimate owner. An obsmarkers can then be
> > automatically created from X¹ to that newer version of X³.
> >
> > >
> > > I /think/ the new visibility work proposed by Jun, Durham, and
> > > others
> > > might offer some solutions to this problem. Rather than speculate
> > > based on my limited knowledge of that proposal, I am hoping someone
> > > with more knowledge could weigh in more definitively.
> > >
> > > It's worth noting that in our proposed workflow, the "integration"
> > > changesets that are rewritten exist in a separate repository that
> > > most people don't pull from. This means we could potentially break
> > > some "rules" about how obsmarkers work since few would notice. But
> > > we
> > > do pull the "integration" repo into the "stable" repo. So
> > > presumably
> > > obsmarkers would propagate to the "stable" repo and be pulled by
> > > people, where they could cause problems.
> > >
> > > Is there a solution to this use case? FWIW, I think a solution
> > > would
> > > have use beyond Mozilla's walls: I'm pretty sure a lot of people
> > > would love for the final history of their repo to be cleaner. It's
> > > just that today's VCS tools (including Git), don't handle
> > > distributed
> > > history rewriting very gracefully, which discourages people from
> > > having nice things.
> >
> >
> > I hope one of the low tech solution offered above solves your
> > immediate problem to let you move forward with your idea.
> >
> > Getting something better than the low-tech solution offered above is
> > going to be a bit harder. The main issues is that we want to
> > actually
> > hide the dropped changeset for all people but some (the
> > author/owner).
> > The definition of that "owner" is not something we have really
> > clarified yet.
> >
> > If we clarifies this ownership things. There are two different
> > approaches we can explore in your case.
> >
> > Server side solution:
> >
> >      Variant of the 'secret' approach. The X² changeset is not served
> > to people unless they are recognized as "owning it" during the
> > discovery phases.
> >
> > Client side solution:
> >
> >      Changeset owned by the local repository, but pruned by a non-
> > owner don't get hidden until something happens (owner actually
> > obsolete
> > them or something; The non-owner pruning could be a flag in the
> > obsmarkers set at pruning time).
>
> By thinking more on the problem, I have a question, does people
> actually needs to pull from the integration repo ? If not, a secret-
> based workflow might do the job.
>

People periodically pull from the integration repo. But we try to
discourage that. CI does need to pull from the integration repo. We can't
easily differentiate CI consumers on the server.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20171113/9bd4cd90/attachment.html>


More information about the Mercurial-devel mailing list