<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>
<div>Accepted and pushed (with Martin's nit fixed). Thanks!</div>
<div>
<div id="MAC_OUTLOOK_SIGNATURE"></div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Martin von Zweigbergk <<a href="mailto:martinvonz@google.com">martinvonz@google.com</a>><br>
<span style="font-weight:bold">Date: </span>Sunday, March 20, 2016 at 8:46 PM<br>
<span style="font-weight:bold">To: </span>Augie Fackler <<a href="mailto:raf@durin42.com">raf@durin42.com</a>>, Durham Goode <<a href="mailto:durham@fb.com">durham@fb.com</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:mercurial-devel@mercurial-scm.org">mercurial-devel@mercurial-scm.org</a>" <<a href="mailto:mercurial-devel@mercurial-scm.org">mercurial-devel@mercurial-scm.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [PATCH remotefilelog-ext] fileserverclient: use new iterbatch() method<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">Nit for Durham to fix in flight: drop the "not" in "isn't streamy not because batch() wasn't"</div>
<span></span><br>
<div class="gmail_quote">
<div dir="ltr">On Sun, Mar 20, 2016, 19:51 Augie Fackler <<a href="mailto:raf@durin42.com" target="_blank">raf@durin42.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
# HG changeset patch<br>
# User Augie Fackler <<a href="mailto:augie@google.com" target="_blank">augie@google.com</a>><br>
# Date 1457036127 18000<br>
# Thu Mar 03 15:15:27 2016 -0500<br>
# Node ID 669622eff4bfc2273a0bd7fd102819fabf0fc03c<br>
# Parent be02547f644b229f84bdeb47970fbb9198741695<br>
fileserverclient: use new iterbatch() method<br>
<br>
This allows the client to send a single batch request for all file contents<br>
and then handle the responses as they stream back to the client, which should<br>
improve both running time and the user experience as far as it goes with<br>
progress.<br>
<br>
diff --git a/remotefilelog/fileserverclient.py b/remotefilelog/fileserverclient.py<br>
--- a/remotefilelog/fileserverclient.py<br>
+++ b/remotefilelog/fileserverclient.py<br>
@@ -6,9 +6,10 @@<br>
# GNU General Public License version 2 or any later version.<br>
<br>
from mercurial.i18n import _<br>
-from mercurial import util, sshpeer, hg, error, util, wireproto, node<br>
+from mercurial import util, sshpeer, hg, error, util, wireproto, node, httppeer<br>
import os, socket, lz4, time, grp, io<br>
import errno<br>
+import itertools<br>
<br>
# Statistics for debugging<br>
fetchcost = 0<br>
@@ -121,6 +122,28 @@ class cacheconnection(object):<br>
<br>
def _getfilesbatch(<br>
remote, receivemissing, progresstick, missed, idmap, batchsize):<br>
+ # Over http(s), iterbatch is a streamy method and we can start<br>
+ # looking at results early. This means we send one (potentially<br>
+ # large) request, but then we show nice progress as we process<br>
+ # file results, rather than showing chunks of $batchsize in<br>
+ # progress.<br>
+ #<br>
+ # Over ssh, iterbatch isn't streamy not because batch() wasn't<br>
+ # explicitly designed as a streaming method. In the future we<br>
+ # should probably introduce a streambatch() method upstream and<br>
+ # use that for this.<br>
+ if (getattr(remote, 'iterbatch', False) and remote.capable('httppostargs')<br>
+ and isinstance(remote, httppeer.httppeer)):<br>
+ b = remote.iterbatch()<br>
+ for m in missed:<br>
+ file_ = idmap[m]<br>
+ node = m[-40:]<br>
+ b.getfile(file_, node)<br>
+ b.submit()<br>
+ for m, r in itertools.izip(missed, b.results()):<br>
+ receivemissing(io.BytesIO('%d\n%s' % (len(r), r)), m)<br>
+ progresstick()<br>
+ return<br>
while missed:<br>
chunk, missed = missed[:batchsize], missed[batchsize:]<br>
b = remote.batch()<br>
_______________________________________________<br>
Mercurial-devel mailing list<br>
<a href="mailto:Mercurial-devel@mercurial-scm.org" target="_blank">Mercurial-devel@mercurial-scm.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mercurial-2Dscm.org_mailman_listinfo_mercurial-2Ddevel&d=CwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=nuarHzhP1wi1T9iURRCj1A&m=hOqR6hvZ019DDGKILl1eQDwGxMGEOnr5xiT7KO5r-Fw&s=a7fqnGtEsFvz0jmVYFIw-1sIJ3lfaLxK6u8fqmKa_NU&e=" rel="noreferrer" target="_blank">https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel</a><br>
</blockquote>
</div>
</div>
</div>
</span>
</body>
</html>