Parallel dirstate.walk?

Matt Mackall mpm at selenic.com
Wed Aug 18 10:45:36 CDT 2010


On Wed, 2010-08-18 at 17:13 +0200, Benoit Boissinot wrote:
> On Wed, Aug 18, 2010 at 5:05 PM, Martin Geisler <mg at aragost.com> wrote:
> > So this is the idea: stat the directories in parallel. I guess this
> > would be done best by replacing dirstate.walk with a osutils.walk that
> > uses C threads internally before it returns a list to its caller. That
> > way the threads will be hidden from sight.
> >
> > Does this sound crazy? Is anybody crazy enough to attempt it? :)
> 
> Not sure it's a win for the non-RAID case (or non-SSD case).
> I'd except it too be highly sequential in the "regular" case (and
> threading could actually hurt).

If everything is cached on the server and there's a non-trivial amount
of network latency then you can get some mileage out of this:

thread 1: request ---------- response
thread 2:          request -------------response

At some point you either saturate the pipe, saturate the client CPU, or
saturate the server CPU. With gigabit ethernet, it's probably the last.

-- 
Mathematics is the supreme nostalgia of our time.




More information about the Mercurial-devel mailing list