Speeding up Mercurial on NFS

Matt Mackall mpm at selenic.com
Mon Dec 6 10:53:09 CST 2010

On Mon, 2010-12-06 at 08:23 -0800, mg at lazybytes.net wrote:
> Hi everybody,
> My client is migrating from ClearCase to Mercurial. Their repositories are
> too big for the individual developers machines, so they want to host
> everything (repository and working copy) on NFS.
> The result is that operations like 'hg status' run much slower than the
> equivalent ClearCase command -- for one repository with 24k files, 'hg
> status' takes 8 seconds. They have some other repositories with 100k
> files.
> Two years ago, someone asked if Git could be made to run faster over NFS.
> The result was a patch that made Git use threads to preload data in
> parallel. This gave a 5 time speedup:
>   http://kerneltrap.org/mailarchive/git/2008/11/14/4089834/thread
> I have tried to replicate these results with some simple test programs:
>   http://bitbucket.org/mg/parallelwalk
> The 'walker' program is a single-threaded C-based program that will walk a
> directory tree as fast as possible, 'pywalker' is a Python-based version
> of it, but with support for multiple threads.

Hmm, to the extent that pywalker spends its time waiting in syscalls, it
might avoid the GIL.

> I have taken the OpenOffice repository and balanced the directories so
> that I have 8 directories of approximately the same size. There are 63k
> files in total. I then ran through these trees using 1-8 pywalker
> processes started in parallel:
>   % timeit -i 1 'pywalker -q dir-1 & ... & pywalker -q dir-8 & wait'
> I also timed the Python threads:
>   % timeit -i 1 'pywalker -t 8 -q dir-[12345678] & wait'
> The results of these tests:
>       processes  threads
>   1:  24 sec     24 sec
>   2:  24 sec     19 sec
>   4:  26 sec     20 sec
>   6:  24 sec     19 sec
>   8:  23 sec     19 sec

What do your C results look like?

> A ~20-25% speedup by using at least two threads in parallel.
> This is on a NFS setup where I export a directory and mount it back on
> localhost. I add an artificial network delay of 0.2 ms (0.1 ms in each
> direction) with
>   % sudo tc qdisc change dev lo root netem delay 0.1ms
> and I empty the disk cache before each run:
>   % sync; sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"

Is this the cache on the client or server?

> Compared to the speedups reported by Git users in the thread I linked
> above, a 25% speedup is very small. Does anybody have any idea for how
> this can be improved?
> While investigating this, I discovered that the NFS protocol has two
> commands for listing filenames in a directory: READDIR and READDIRPLUS.
> The latter will also return stat information for each file found -- this
> saves a lot of round-trips. Unfortunately, I saw the kernel use
> READDIRPLUS only once in a while -- this is a shame since the speed
> advantage of READDIRPLUS is huge. Does anybody know if it is possible for
> a user-space program like Mercurial to influence how the kernel chooses
> each operation?

A good question. You should try tracing ls(1) or find(1) and see if it's
doing the right thing for starters. If not, you're probably screwed.

Mathematics is the supreme nostalgia of our time.

More information about the Mercurial-devel mailing list