[issue2188] Operations Fail with 404 with lots of repos in hgweb.config and small RPS load

[Matt Hawley]

New submission from Matt Hawley <matt.hawley at microsoft.com>:

In a hosted environment that has lots of Hg repositories (CodePlex) we are 
running into issues where about 10-12 Requests Per Second to the server 
will result in 404 errors on simple operations (as well as cause pushes to 
fail with 500 errors). During our performance testing, we identified that 
setting up the hgweb.config file as

[paths]
/ = \\repo\path\*

Was very inneficient as the number of repositories grows the time taken 
for operations wish linearly getting worse (we were seeing about 90 
seconds to pull/push with 5000 repositories). Looking at the source during 
that time, indicated that using the wildcard scans the file system to 
generate a list of repositories. Our solution for this performance problem 
was to use the following format in the hgweb.config file

[paths]
/repo1 = \\repo\path\repo1
/repo2 = \\repo\path\repo2
...
/repoN = \\repo\path\repoN

However, now that our hgweb.config file has grown to a decent size (we 
have 1000+ repositories hosted), we start seeing 404 errors reported by 
hgweb under the load previously mentioned. Looking at the performance 
monitors on the server shows that we are not CPU bound (it hovers <15%). 

The only thing I can assume, is that reading of the hgweb.config each time 
sequentially is causing the server to not be able to serve the requests 
fast enough. Because of this, our users are experiencing problems when 
accessing their repositories.

Our setup and performance testing has confirmed this to be the case when 
hosted in IIS 7.5 with both hgwebdir.cgi and ISAPI-WSGI hosting.

----------
messages: 12498
nosy: matthawley
priority: urgent
status: unread
title: Operations Fail with 404 with lots of repos in hgweb.config and small RPS load
topic: hgweb

____________________________________________________
Mercurial issue tracker <bugs at mercurial.selenic.com>
<http://mercurial.selenic.com/bts/issue2188>
____________________________________________________