[PATCH 08 of 10 shelve-ext v3] shelve: add obs-based unshelve functionality

[Jun Wu]

Excerpts from Sean Farley's message of 2017-02-06 11:41:55 -0800:
> Perhaps I'm the only one worried about adding all these markers. To me,
> it would make more sense to delete these markers since we're never going
> to push these shelved changesets.

You are not the only one.

While I agree that the perf of obsstore is a pain, all possible fixes that I
can think of require too much work that should not block this series. Let's
talk about it during the sprint, or earlier if you want.

I'd like to note that the traditional "strip" is while useful, not friendly
to a giant repo - it's slow and dangerous.

I personally like to see a data structure similar with "sparkey" [1]
implemented in core. The data structure has two parts:

  - data:  append-only, include deleting operations (like a journal)
  - index: NOT append-only; do not include pointers to deleted data
           could be rebuilt from data

For the index part, sparkey uses a hash-table. We may or may not want a
git-packfile-index-like thing, depending if the "key"s are all SHA1s or not.

That would be useful to all kinds of "quick-random-lookups" without
loading/building the whole index problems, although some of them could be
worked around by the chg repo preloading plan.

[1]: https://github.com/spotify/sparkey

> Also, this sounds like we're overloading the concept of markers. In the
> past, I have wanted a concept of "markers that hide changesets" but to
> group them into different namespaces. For example, remote branches not
> checked out locally could be hidden into one such group, shelved changes
> into another group, etc.

That sounds related to Stanislau's bitmap approach. "hidden" is not
necessarily implemented as obsmarkers. They can come from other sources:

  obsstore -> obsstore hidden bitmap -+
                                      |
              shelve hidden bitmap   -+- (bitmap or) -> final hidden bitmap
                                      |
              other hidden bitmap    -+

(cc Stanislau)

> Though, I do realize I might be derailing this topic by trying to
> generalize this.