Speeding up Mercurial on NFS

mg at lazybytes.net mg at lazybytes.net
Mon Dec 6 10:23:39 CST 2010


Hi everybody,

My client is migrating from ClearCase to Mercurial. Their repositories are
too big for the individual developers machines, so they want to host
everything (repository and working copy) on NFS.

The result is that operations like 'hg status' run much slower than the
equivalent ClearCase command -- for one repository with 24k files, 'hg
status' takes 8 seconds. They have some other repositories with 100k
files.

Two years ago, someone asked if Git could be made to run faster over NFS.
The result was a patch that made Git use threads to preload data in
parallel. This gave a 5 time speedup:

  http://kerneltrap.org/mailarchive/git/2008/11/14/4089834/thread


I have tried to replicate these results with some simple test programs:

  http://bitbucket.org/mg/parallelwalk

The 'walker' program is a single-threaded C-based program that will walk a
directory tree as fast as possible, 'pywalker' is a Python-based version
of it, but with support for multiple threads. I made these simple programs
to establish an upper bound on the possible speedup -- if I see a speedup
of 300% using the simple programs, then the final speedup will surely be
less because of all the extra computations dirstate.py goes through.

I have taken the OpenOffice repository and balanced the directories so
that I have 8 directories of approximately the same size. There are 63k
files in total. I then ran through these trees using 1-8 pywalker
processes started in parallel:

  % timeit -i 1 'pywalker -q dir-1 & ... & pywalker -q dir-8 & wait'

I also timed the Python threads:

  % timeit -i 1 'pywalker -t 8 -q dir-[12345678] & wait'

The results of these tests:

      processes  threads
  1:  24 sec     24 sec
  2:  24 sec     19 sec
  4:  26 sec     20 sec
  6:  24 sec     19 sec
  8:  23 sec     19 sec

A ~20-25% speedup by using at least two threads in parallel.

This is on a NFS setup where I export a directory and mount it back on
localhost. I add an artificial network delay of 0.2 ms (0.1 ms in each
direction) with

  % sudo tc qdisc change dev lo root netem delay 0.1ms

and I empty the disk cache before each run:

  % sync; sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"


Compared to the speedups reported by Git users in the thread I linked
above, a 25% speedup is very small. Does anybody have any idea for how
this can be improved?

While investigating this, I discovered that the NFS protocol has two
commands for listing filenames in a directory: READDIR and READDIRPLUS.
The latter will also return stat information for each file found -- this
saves a lot of round-trips. Unfortunately, I saw the kernel use
READDIRPLUS only once in a while -- this is a shame since the speed
advantage of READDIRPLUS is huge. Does anybody know if it is possible for
a user-space program like Mercurial to influence how the kernel chooses
each operation?

Please let me know if you have any good ideas in this area!

-- 
Martin Geisler



More information about the Mercurial-devel mailing list