Design document for allowing alternative client side storage

[Gregory Szorc]

On Thu, Mar 10, 2016 at 5:46 PM, Durham Goode <durham at fb.com> wrote:

> We're going to be investigating alternative client side storage strategies
> for Mercurial at Facebook over the next few months. We've already moved off
> revlogs for our filelog storage (via remotefilelog), and will likely need
> to avoid revlogs when we move to tree manifests as well.
>
> As part of this, I've put together a design doc describing a high level
> idea that would let us experiment with different storage backends, and
> provides a path for migrating existing users over. It's currently focused
> on situations like ours, where you have parts of the repository on a
> central server and parts on the client, but the overall design may be of
> interest to the community.
>
> https://quip.com/TFR2Aw0lu0LB
>
> It's a bit light on concrete format details, since the main goal is to put
> abstractions in place that would let us break away from the existing
> formats and experiment.
>
> Feel free to comment on the doc (you have to sign in to Quip to comment),
> or respond by email.
>

Interesting read. My knee jerk takeaway is it is similar to my
PackedRepoPlan but with a formal separate metadata store. However, you say
in this document that "This means the metadata store could be recomputed
from the content store if necessary." I translate this to mean the metadata
store is a glorified cache. When you view the metadata store as a cache,
the high-level proposals in this doc are compatible with ideas in
PackedRepoPlan.

Could I bother you to add a section on transactions and reader/writer
consistency to the document? As it stands, I have some questions on how
you'll work consistent views/snapshots into this document. Specifically,
performing transactions across multiple systems (such as SQLite) could be
problematic. The scenario I'm envisioning is one where the content and
metadata stores are in separate stores that both support transactions. You
can commit a transaction in 1 but not the other. How do you roll back the
first committed transaction and/or provide a consistent reader view over
the union of the 2 stores? With SQLite, you can make a copy of the
self-contained database file and move it back to restore previous state.
But with most hosted stores (like MySQL), I don't think you can roll back
the last transaction once it has been committed. I think very few stores
will support this property (basically limited to things running on a POSIX
filesystem). If very few stores support that property, does a high-level
abstraction of content and metadata stores make sense or should we be
focusing on a concrete implementation [that relies on POSIX filesystem
semantics]?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160310/4baf4f0d/attachment.html>