Design document for allowing alternative client side storage

[Durham Goode]

On 3/10/16 8:27 PM, Gregory Szorc wrote:
> On Thu, Mar 10, 2016 at 5:46 PM, Durham Goode <durham at fb.com 
> <mailto:durham at fb.com>> wrote:
>
>     We're going to be investigating alternative client side storage
>     strategies for Mercurial at Facebook over the next few months.
>     We've already moved off revlogs for our filelog storage (via
>     remotefilelog), and will likely need to avoid revlogs when we move
>     to tree manifests as well.
>
>     As part of this, I've put together a design doc describing a high
>     level idea that would let us experiment with different storage
>     backends, and provides a path for migrating existing users over.
>     It's currently focused on situations like ours, where you have
>     parts of the repository on a central server and parts on the
>     client, but the overall design may be of interest to the community.
>
>     https://quip.com/TFR2Aw0lu0LB
>
>     It's a bit light on concrete format details, since the main goal
>     is to put abstractions in place that would let us break away from
>     the existing formats and experiment.
>
>     Feel free to comment on the doc (you have to sign in to Quip to
>     comment), or respond by email.
>
>
> Interesting read. My knee jerk takeaway is it is similar to my 
> PackedRepoPlan but with a formal separate metadata store.
I think the PackedRepo plan is a potential implementation of the long 
term storage I mention. The overall plan encompasses other things like: 
the removal of the concept of rev numbers, a potentially storage 
agnostic concept of repacking, a division of server fetched and local 
created data (and implying a way of fetching them), and the metadata 
separation you mention.
> However, you say in this document that "This means the metadata store 
> could be recomputed from the content store if necessary." I translate 
> this to mean the metadata store is a glorified cache. When you view 
> the metadata store as a cache, the high-level proposals in this doc 
> are compatible with ideas in PackedRepoPlan.
It's not quite a cache.  A big reason to separate the metadata is so we 
can store the metadata without storing the revision content at all.
>
> Could I bother you to add a section on transactions and reader/writer 
> consistency to the document? As it stands, I have some questions on 
> how you'll work consistent views/snapshots into this document. 
> Specifically, performing transactions across multiple systems (such as 
> SQLite) could be problematic. The scenario I'm envisioning is one 
> where the content and metadata stores are in separate stores that both 
> support transactions. You can commit a transaction in 1 but not the 
> other. How do you roll back the first committed transaction and/or 
> provide a consistent reader view over the union of the 2 stores? With 
> SQLite, you can make a copy of the self-contained database file and 
> move it back to restore previous state. But with most hosted stores 
> (like MySQL), I don't think you can roll back the last transaction 
> once it has been committed. I think very few stores will support this 
> property (basically limited to things running on a POSIX filesystem). 
> If very few stores support that property, does a high-level 
> abstraction of content and metadata stores make sense or should we be 
> focusing on a concrete implementation [that relies on POSIX filesystem 
> semantics]?
This is one area where we'll likely deviate from the normal Mercurial 
patterns.  I anticipate our store being explicitly append only (in terms 
of only adding data, not in terms of appending to an end of a file).  It 
will be entirely content addressed, so any data that is added can be 
permanent (or cleaned up in a rare mark-sweep gc-style clean up). So 
there will be no rollback, and no need for rollback.

When data is added, I imagine it being done either in a third-party 
transacted way (ala sqlite, where many writers can be coordinated) or in 
a posix way (ala git, where many writers just operate on different 
files, and compaction happens by reading immutable files and producing 
immutable files).  In this way, we could allow multiple writers at once.

If we need to support something like the changelog, where there needs to 
be the concept of deleting branches of the tree, then we'll create a 
separate store of branch heads, and treat them like git refs (where the 
branch refs determine what parts of the key/value store are accessible). 
Then the concept of a consistent view just boils down to ensuring we've 
written data to the append only store before we update the branch heads 
list.

Does that answer your question? If so, I can add it to the document.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160311/448fda41/attachment.html>