[PATCH remotefilelog-ext] fileserverclient: use new iterbatch() method

Martin von Zweigbergk martinvonz at google.com
Sun Mar 20 23:46:42 EDT 2016


Nit for Durham to fix in flight: drop the "not" in "isn't streamy not
because batch() wasn't"

On Sun, Mar 20, 2016, 19:51 Augie Fackler <raf at durin42.com> wrote:

> # HG changeset patch
> # User Augie Fackler <augie at google.com>
> # Date 1457036127 18000
> #      Thu Mar 03 15:15:27 2016 -0500
> # Node ID 669622eff4bfc2273a0bd7fd102819fabf0fc03c
> # Parent  be02547f644b229f84bdeb47970fbb9198741695
> fileserverclient: use new iterbatch() method
>
> This allows the client to send a single batch request for all file contents
> and then handle the responses as they stream back to the client, which
> should
> improve both running time and the user experience as far as it goes with
> progress.
>
> diff --git a/remotefilelog/fileserverclient.py
> b/remotefilelog/fileserverclient.py
> --- a/remotefilelog/fileserverclient.py
> +++ b/remotefilelog/fileserverclient.py
> @@ -6,9 +6,10 @@
>  # GNU General Public License version 2 or any later version.
>
>  from mercurial.i18n import _
> -from mercurial import util, sshpeer, hg, error, util, wireproto, node
> +from mercurial import util, sshpeer, hg, error, util, wireproto, node,
> httppeer
>  import os, socket, lz4, time, grp, io
>  import errno
> +import itertools
>
>  # Statistics for debugging
>  fetchcost = 0
> @@ -121,6 +122,28 @@ class cacheconnection(object):
>
>  def _getfilesbatch(
>          remote, receivemissing, progresstick, missed, idmap, batchsize):
> +    # Over http(s), iterbatch is a streamy method and we can start
> +    # looking at results early. This means we send one (potentially
> +    # large) request, but then we show nice progress as we process
> +    # file results, rather than showing chunks of $batchsize in
> +    # progress.
> +    #
> +    # Over ssh, iterbatch isn't streamy not because batch() wasn't
> +    # explicitly designed as a streaming method. In the future we
> +    # should probably introduce a streambatch() method upstream and
> +    # use that for this.
> +    if (getattr(remote, 'iterbatch', False) and
> remote.capable('httppostargs')
> +        and isinstance(remote, httppeer.httppeer)):
> +        b = remote.iterbatch()
> +        for m in missed:
> +            file_ = idmap[m]
> +            node = m[-40:]
> +            b.getfile(file_, node)
> +        b.submit()
> +        for m, r in itertools.izip(missed, b.results()):
> +            receivemissing(io.BytesIO('%d\n%s' % (len(r), r)), m)
> +            progresstick()
> +        return
>      while missed:
>          chunk, missed = missed[:batchsize], missed[batchsize:]
>          b = remote.batch()
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160321/3d9bdae6/attachment.html>


More information about the Mercurial-devel mailing list