[Bug 4735] New: Large regression in ssh download times

mercurial-bugs at selenic.com mercurial-bugs at selenic.com
Fri Jun 26 07:05:36 CDT 2015


          Priority: urgent
            Bug ID: 4735
                CC: mercurial-devel at selenic.com,
                    pierre-yves.david at ens-lyon.org
          Assignee: bugzilla at selenic.com
           Summary: Large regression in ssh download times
          Severity: bug
    Classification: Unclassified
                OS: Mac OS
          Reporter: durham at fb.com
          Hardware: PC
            Status: UNCONFIRMED
           Version: default branch
         Component: Mercurial
           Product: Mercurial

The per 'line' download time for an ssh download using the sshpeer seems to
have become exponentially worse.  Downloading a 100MB file via sshpeer went
from 3s to 67s and downloading larger files seems to be way way worse. This
mainly affects the external remotefilelog extension, but may also affect clone
times for repos with large files (haven't tested)

It appears to be cpu bound on the client side. My theory is that
bufferedinputpipe._lenbuffer is n^2 since it recomputes every buffer packet's
length, for every packet.

The a partial profile is as follows:
~/foo> hg prefetch -r . --profile
] 0/1^
C| 94.8%  fileserverclient.py: prefetch             line 132: 
 \ 68.1%  fileserverclient.py: request             line 302:  missingids =
   | 68.0%  fileserverclient.py: receivemissing    line 196: 
   | 68.0%  sshpeer.py:     read                   line 223:  data =
   | 68.0%  sshpeer.py:     _call                  line 82:  return
self._call('read', s...
   | 68.0%  util.py:        read                   line 103:  return meth(data)
   | 67.0%  util.py:        _lenbuf                line 277:  while (not
self._eof) and (...
   | 66.5%  util.py:        <genexpr>              line 304:  return sum(len(d)
for d in ...
 \  9.1%  fileserverclient.py: __contains__        line 290:  if cachekey in
   |  6.5%  genericpath.py: exists                 line 333:  exists =
 \  6.7%  fileserverclient.py: getcachekey         line 287:  cachekey =
Sample count: 772
Total time: 17.350000 seconds

You are receiving this mail because:
You are on the CC list for the bug.

More information about the Mercurial-devel mailing list