Speeding up Mercurial on NFS

[Martin Geisler]

Matt Mackall <mpm at selenic.com> writes:

> On Mon, 2010-12-06 at 08:23 -0800, mg at lazybytes.net wrote:
>
>> Two years ago, someone asked if Git could be made to run faster over
>> NFS. The result was a patch that made Git use threads to preload data
>> in parallel. This gave a 5 time speedup:
>>
>>   http://kerneltrap.org/mailarchive/git/2008/11/14/4089834/thread
>>
>> I have tried to replicate these results with some simple test
>> programs:
>>
>>   http://bitbucket.org/mg/parallelwalk
>>
>> The 'walker' program is a single-threaded C-based program that will
>> walk a directory tree as fast as possible, 'pywalker' is a
>> Python-based version of it, but with support for multiple threads.
>
> Hmm, to the extent that pywalker spends its time waiting in syscalls,
> it might avoid the GIL.

Yes, that's the idea: each thread releases the GIL (I checked the Python
source) when it does os.lstat and lets another thread run Python byte
code until that thread hits os.lstat.

>> The results of these tests:
>>
>>       processes  threads
>>   1:  24 sec     24 sec
>>   2:  24 sec     19 sec
>>   4:  26 sec     20 sec
>>   6:  24 sec     19 sec
>>   8:  23 sec     19 sec
>
> What do your C results look like?

I only have a single threaded C program, and it was about 30% faster
than the Python program, if I recall correctly. I can test on the office
machine tomorrow.

>> This is on a NFS setup where I export a directory and mount it back
>> on localhost. I add an artificial network delay of 0.2 ms (0.1 ms in
>> each direction) with
>>
>>   % sudo tc qdisc change dev lo root netem delay 0.1ms
>>
>> and I empty the disk cache before each run:
>>
>>   % sync; sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
>
> Is this the cache on the client or server?

Sort of both -- I'm using the same machine as client and server with the
NFS export mounted back on localhost.

>> While investigating this, I discovered that the NFS protocol has two
>> commands for listing filenames in a directory: READDIR and
>> READDIRPLUS. The latter will also return stat information for each
>> file found -- this saves a lot of round-trips. Unfortunately, I saw
>> the kernel use READDIRPLUS only once in a while -- this is a shame
>> since the speed advantage of READDIRPLUS is huge. Does anybody know
>> if it is possible for a user-space program like Mercurial to
>> influence how the kernel chooses each operation?
>
> A good question. You should try tracing ls(1) or find(1) and see if
> it's doing the right thing for starters. If not, you're probably
> screwed.

Yeah, I think I'm screwed... I see no clear pattern as to when the
kernel issues a READDIR call and when it uses READDIRPLUS. I tried
reading the code in fs/nfs/ in the Linux kernel, and in inode.c I found
a constant that limits READDIRPLUS:

  /* Don't use READDIRPLUS on directories that we believe are too large */
  #define NFS_LIMIT_READDIRPLUS (8*PAGE_SIZE)

Apart from that, I got the impression that READDIRPLUS is used as much
as possible. But when I run ls(1) or find(1) repeatedly, it is only used
sometimes.

My guess is that READDIRPLUS is used when opendir() is called. However,
directories are cached with a timeout of 30 seconds by default and files
are only cached for 3 seconds by default -- so on the second run, stat
information for the directories may well be cached whereas the file stat
info has expired => a GETATTR call per file instead of a few READDIRPLUS
calls.

-- 
Martin Geisler

Mercurial links: http://mercurial.ch/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://selenic.com/pipermail/mercurial-devel/attachments/20101207/05863c54/attachment.pgp>