Design document for allowing alternative client side storage

Durham Goode durham at fb.com
Thu Mar 10 21:53:51 EST 2016



On 3/10/16 6:13 PM, Sean Farley wrote:
> Sean Farley <sean at farley.io> writes:
>
>> Durham Goode <durham at fb.com> writes:
>>
>>> We're going to be investigating alternative client side storage
>>> strategies for Mercurial at Facebook over the next few months. We've
>>> already moved off revlogs for our filelog storage (via remotefilelog),
>>> and will likely need to avoid revlogs when we move to tree manifests as
>>> well.
>>>
>>> As part of this, I've put together a design doc describing a high level
>>> idea that would let us experiment with different storage backends, and
>>> provides a path for migrating existing users over. It's currently
>>> focused on situations like ours, where you have parts of the repository
>>> on a central server and parts on the client, but the overall design may
>>> be of interest to the community.
>>>
>>> https://quip.com/TFR2Aw0lu0LB
>>>
>>> It's a bit light on concrete format details, since the main goal is to
>>> put abstractions in place that would let us break away from the existing
>>> formats and experiment.
>>>
>>> Feel free to comment on the doc (you have to sign in to Quip to
>>> comment), or respond by email.
>> It'd be interesting if this was modular enough so that the server could
>> offload it's storage to a CDN (or S3 or whatever).
One of the goals is to enable look-aside revision fetching (like 
remotefilelog already has, where we fetch file contents from memcache) 
for all contents of the repository.  So yea, from a client side this is 
a property we'll be aiming for. But it's more a side effect of going 
with a purely key/value based storage (instead of globally order revs). 
Not so much a side effect of the modular-ness.
> For more context: trying to replace the need for largefiles.
Yea, it could help.  But there's still the problem of mercurial thinking 
it can load every file completely into memory, so it doesn't help the 
multi-GB large file problem so much.


More information about the Mercurial-devel mailing list