Design document for allowing alternative client side storage
durham at fb.com
Fri Mar 11 04:02:03 EST 2016
On 3/10/16 8:27 PM, Gregory Szorc wrote:
> On Thu, Mar 10, 2016 at 5:46 PM, Durham Goode <durham at fb.com
> <mailto:durham at fb.com>> wrote:
> We're going to be investigating alternative client side storage
> strategies for Mercurial at Facebook over the next few months.
> We've already moved off revlogs for our filelog storage (via
> remotefilelog), and will likely need to avoid revlogs when we move
> to tree manifests as well.
> As part of this, I've put together a design doc describing a high
> level idea that would let us experiment with different storage
> backends, and provides a path for migrating existing users over.
> It's currently focused on situations like ours, where you have
> parts of the repository on a central server and parts on the
> client, but the overall design may be of interest to the community.
> It's a bit light on concrete format details, since the main goal
> is to put abstractions in place that would let us break away from
> the existing formats and experiment.
> Feel free to comment on the doc (you have to sign in to Quip to
> comment), or respond by email.
> Interesting read. My knee jerk takeaway is it is similar to my
> PackedRepoPlan but with a formal separate metadata store.
I think the PackedRepo plan is a potential implementation of the long
term storage I mention. The overall plan encompasses other things like:
the removal of the concept of rev numbers, a potentially storage
agnostic concept of repacking, a division of server fetched and local
created data (and implying a way of fetching them), and the metadata
separation you mention.
> However, you say in this document that "This means the metadata store
> could be recomputed from the content store if necessary." I translate
> this to mean the metadata store is a glorified cache. When you view
> the metadata store as a cache, the high-level proposals in this doc
> are compatible with ideas in PackedRepoPlan.
It's not quite a cache. A big reason to separate the metadata is so we
can store the metadata without storing the revision content at all.
> Could I bother you to add a section on transactions and reader/writer
> consistency to the document? As it stands, I have some questions on
> how you'll work consistent views/snapshots into this document.
> Specifically, performing transactions across multiple systems (such as
> SQLite) could be problematic. The scenario I'm envisioning is one
> where the content and metadata stores are in separate stores that both
> support transactions. You can commit a transaction in 1 but not the
> other. How do you roll back the first committed transaction and/or
> provide a consistent reader view over the union of the 2 stores? With
> SQLite, you can make a copy of the self-contained database file and
> move it back to restore previous state. But with most hosted stores
> (like MySQL), I don't think you can roll back the last transaction
> once it has been committed. I think very few stores will support this
> property (basically limited to things running on a POSIX filesystem).
> If very few stores support that property, does a high-level
> abstraction of content and metadata stores make sense or should we be
> focusing on a concrete implementation [that relies on POSIX filesystem
This is one area where we'll likely deviate from the normal Mercurial
patterns. I anticipate our store being explicitly append only (in terms
of only adding data, not in terms of appending to an end of a file). It
will be entirely content addressed, so any data that is added can be
permanent (or cleaned up in a rare mark-sweep gc-style clean up). So
there will be no rollback, and no need for rollback.
When data is added, I imagine it being done either in a third-party
transacted way (ala sqlite, where many writers can be coordinated) or in
a posix way (ala git, where many writers just operate on different
files, and compaction happens by reading immutable files and producing
immutable files). In this way, we could allow multiple writers at once.
If we need to support something like the changelog, where there needs to
be the concept of deleting branches of the tree, then we'll create a
separate store of branch heads, and treat them like git refs (where the
branch refs determine what parts of the key/value store are accessible).
Then the concept of a consistent view just boils down to ensuring we've
written data to the append only store before we update the branch heads
Does that answer your question? If so, I can add it to the document.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Mercurial-devel