D7882: lfs: avoid quadratic performance in processing server responses

[mharbison72 (Matt Harbison)]

mharbison72 created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  This is also adapted from the Facebook repo[1].  Unlike there, we were already
  reading the download stream in chunks and immediately writing it to disk, so we
  basically avoided the problem on download.  There shouldn't be a lot of data to
  read on upload, but it's better to get rid of this pattern.
  
  [1] https://github.com/facebookexperimental/eden/commit/82df66ffe97e21f3ee73dfec093c87500fc1f6a7

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D7882

AFFECTED FILES
  hgext/lfs/blobstore.py

CHANGE DETAILS

diff --git a/hgext/lfs/blobstore.py b/hgext/lfs/blobstore.py
--- a/hgext/lfs/blobstore.py
+++ b/hgext/lfs/blobstore.py
@@ -503,7 +503,6 @@
         for k, v in headers:
             request.add_header(pycompat.strurl(k), pycompat.strurl(v))
 
-        response = b''
         try:
             with contextlib.closing(self.urlopener.open(request)) as res:
                 contentlength = res.info().get(b"content-length")
@@ -520,11 +519,14 @@
                     # blobstore
                     localstore.download(oid, res, contentlength)
                 else:
+                    blocks = []
                     while True:
                         data = res.read(1048576)
                         if not data:
                             break
-                        response += data
+                        blocks.append(data)
+
+                    response = b"".join(blocks)
                     if response:
                         ui.debug(b'lfs %s response: %s' % (action, response))
         except util.urlerr.httperror as ex:



To: mharbison72, #hg-reviewers
Cc: mercurial-devel