When pushing a largefiles repo with N largefiles, we make N requests to see if each file exists on the server. We should batch these into a single request. For the N' largefiles that are missing from the server, we do N' uploads. We should batch these, probably with a multi-part POST. The per-request overhead can add up substantially, especially when SSL handshakes are involved.
Great point! I remember Lantiq saying that the overhead in bfiles (and hence largefiles) was one of the reasons they began writing the snap extension. Apparently they had some 20,000 files, each 10 MB in size, and even a 100 ms overhead per file will then make 'hg push' take 40 minuttes extra (even if nothing needs to be uploaded).
This is partially related to issue3160 The general plan has been that we want to implement an http connection pool and use that instead of what we're doing now but, no one has done it yet. The problem could also be fixed with what Kevin describes below, but of course someone needs to do it :-P Na'Tosha
The current wire protocol code already supports request batching generically.
--- Bug imported by bugzilla@serpentine.com 2012-05-12 09:30 EDT --- This bug was previously known as _bug_ 3385 at http://mercurial.selenic.com/bts/issue3385
Patch for batching the statlfile command submitted to the list here: http://www.selenic.com/pipermail/mercurial-devel/2012-June/042013.html
Fixed in stable by: 9e1616307c4c largefiles: batch statlfile requests when pushing a largefiles repo
Mass close old bugs in testing.