[PATCH 01 of 21 V2] hgext: `speedy` - the history query accelerator

[Augie Fackler]

On Dec 28, 2012, at 6:16 PM, Matt Mackall <mpm at selenic.com> wrote:

> On Thu, 2012-12-13 at 18:52 -0800, Tomasz Kleczek wrote:
>> # HG changeset patch
>> # User Tomasz Kleczek <tkleczek at fb.com>
>> # Date 1355434337 28800
>> # Node ID 136696ee699f3cfa85fd9e43a95f3ec9f1bc1c74
>> # Parent  2b79504469cd9448e0e2768acd1a057a527d30aa
>> hgext: `speedy` - the history query accelerator
> 
> So I've been kicking this and related ideas around for a while and my
> conclusion is that this is not nearly ambitious enough, especially for a
> Facebook-scale deployment.
> 
> Instead, I think you need to be thinking in terms of a service that also
> saves you most of the cost of cloning and pulling... by also managing
> storage!
> 
> That probably sounds like heresy: isn't the whole point of a distributed
> SCM to have no central server? Well, not exactly. The point is really to
> have a workflow where synchronization is optional, where working
> detached is possible, and no authority is needed to commit. If done
> right (ie not monolithic CVS/SVN/Perforce style), you can have a
> distributed service that's faster than local disk and scales to
> thousands of users.
> 
> Google's implementation of a bigtable-based hg backend for Google Code
> is a decent proof-of-concept of this. But we should be able to build a
> service that plugs into any key/value store (be it bigtable, hadoop,
> MySql, etc.) and/or memory cache service with just a little glue.

I'd be happy to talk to folks about this, and try and help with the design. Ideally, we'd come up with something that'd make the bigtable-backed design easier to support in the future.

> But I think we need to bang out what that looks like a bit more before
> committing to including/supporting something like this implementation.

I like the idea of an index server for doing revset queries - I've actually been hoping to try and build something along those lines in a generic way to have a stupidly fast history browser, but I've been lacking enough round tuits to make that happen.

I think decoupling the revset index from the actual repository storage layer feels right, but I could be convinced that's crazy.

> 
> -- 
> Mathematics is the supreme nostalgia of our time.
> 
> 
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at selenic.com
> http://selenic.com/mailman/listinfo/mercurial-devel