[PATCH 3 of 3 V2] exchange: make clone bundles non-experimental and enabled by default

Augie Fackler raf at durin42.com
Tue Jan 12 09:12:30 CST 2016


On Fri, Jan 08, 2016 at 11:02:21AM -0800, Gregory Szorc wrote:
> # HG changeset patch
> # User Gregory Szorc <gregory.szorc at gmail.com>
> # Date 1452279484 28800
> #      Fri Jan 08 10:58:04 2016 -0800
> # Node ID 31875bdd9fc69b4c68c2714ecc9c4a98a9e2275b
> # Parent  074fc9366cbb65fd16d449bdc0f372317142b76a
> exchange: make clone bundles non-experimental and enabled by default

Queued! Thanks for taking my hacky proof of concept and turning it in
to something useful.

>
> The clone bundles feature was introduced in Mercurial 3.6 behind an
> experimental and disabled by default flag. The feature has been enabled
> on hg.mozilla.org for a few months and has served many terabytes of
> clones. Users have been encouraged to use the feature and reception
> has been very positive (mainly due to faster clones as a result of
> connecting to a CDN). I have heard no feedback about changing the
> feature other than inquiries about when it will be enabled by default.
> So, I think the feature is ready to be enabled by default.
>
> This patch renames experimental.clonebundles to ui.clonebundles,
> documents the option, and enables it by default. References to the
> experimental state of clone bundles have been removed. The remaining
> config option docs in clonebundles.py have been removed because they
> are redudant with `hg help config`.
>
> There are some oddities with behavior of clone bundles. Because clones
> with clone bundles are effectively 2 `hg pull` operations, there may be
> 2 transactions. This could result in hooks running twice. If the
> subsequent pull is aborted, it could result in partial rollback and an
> incomplete clone. This behavior is a bit wonky and should probably
> be documented. If this patch is accepted, I'll send a follow-up to
> document it. I don't think this behavior should prevent the feature
> being enabled by default. Reworking the clone mechanism to support
> interrupted or multi-part clones feels like a major new feature and
> something that when implemented can change the hook and rollback
> semantics of clone bundles. Besides, partial clone is better than
> full rollback and hooks running on initial clone are likely rare, so I
> think the impact is minimal.
>
> diff --git a/hgext/clonebundles.py b/hgext/clonebundles.py
> --- a/hgext/clonebundles.py
> +++ b/hgext/clonebundles.py
> @@ -1,12 +1,12 @@
>  # This software may be used and distributed according to the terms of the
>  # GNU General Public License version 2 or any later version.
>
> -"""advertise pre-generated bundles to seed clones (experimental)
> +"""advertise pre-generated bundles to seed clones
>
>  "clonebundles" is a server-side extension used to advertise the existence
>  of pre-generated, externally hosted bundle files to clients that are
>  cloning so that cloning can be faster, more reliable, and require less
>  resources on the server.
>
>  Cloning can be a CPU and I/O intensive operation on servers. Traditionally,
>  the server, in response to a client's request to clone, dynamically generates
> @@ -155,26 +155,16 @@ Because there is no automatic Mercurial
>  bundle hosting service, it is important for server operators to view the bundle
>  hosting service as an extension of the Mercurial server in terms of
>  availability and service level agreements: if the bundle hosting service goes
>  down, so does the ability for clients to clone. Note: clients will see a
>  message informing them how to bypass the clone bundles facility when a failure
>  occurs. So server operators should prepare for some people to follow these
>  instructions when a failure occurs, thus driving more load to the original
>  Mercurial server when the bundle hosting service fails.
> -
> -The following config options influence the behavior of the clone bundles
> -feature:
> -
> -ui.clonebundlefallback
> -   Whether to automatically fall back to a traditional clone in case of
> -   clone bundles failure. Defaults to false for reasons described above.
> -
> -experimental.clonebundles
> -   Whether the clone bundles feature is enabled on clients. Defaults to true.
>  """
>
>  from mercurial import (
>      extensions,
>      wireproto,
>  )
>
>  testedwith = 'internal'
> diff --git a/mercurial/exchange.py b/mercurial/exchange.py
> --- a/mercurial/exchange.py
> +++ b/mercurial/exchange.py
> @@ -1670,17 +1670,17 @@ def unbundle(repo, cg, heads, source, ur
>      return r
>
>  def _maybeapplyclonebundle(pullop):
>      """Apply a clone bundle from a remote, if possible."""
>
>      repo = pullop.repo
>      remote = pullop.remote
>
> -    if not repo.ui.configbool('experimental', 'clonebundles', False):
> +    if not repo.ui.configbool('ui', 'clonebundles', True):
>          return
>
>      # Only run if local repo is empty.
>      if len(repo):
>          return
>
>      if pullop.heads:
>          return
> @@ -1726,17 +1726,17 @@ def _maybeapplyclonebundle(pullop):
>      # clone load to be offloaded.
>      elif repo.ui.configbool('ui', 'clonebundlefallback', False):
>          repo.ui.warn(_('falling back to normal clone\n'))
>      else:
>          raise error.Abort(_('error applying bundle'),
>                            hint=_('if this error persists, consider contacting '
>                                   'the server operator or disable clone '
>                                   'bundles via '
> -                                 '"--config experimental.clonebundles=false"'))
> +                                 '"--config ui.clonebundles=false"'))
>
>  def parseclonebundlesmanifest(repo, s):
>      """Parses the raw text of a clone bundles manifest.
>
>      Returns a list of dicts. The dicts have a ``URL`` key corresponding
>      to the URL and other keys are the attributes for the entry.
>      """
>      m = []
> diff --git a/mercurial/help/config.txt b/mercurial/help/config.txt
> --- a/mercurial/help/config.txt
> +++ b/mercurial/help/config.txt
> @@ -1482,16 +1482,26 @@ User interface controls.
>
>  ``askusername``
>      Whether to prompt for a username when committing. If True, and
>      neither ``$HGUSER`` nor ``$EMAIL`` has been specified, then the user will
>      be prompted to enter a username. If no username is entered, the
>      default ``USER at HOST`` is used instead.
>      (default: False)
>
> +``clonebundles``
> +    Whether the "clone bundles" feature is enabled.
> +
> +    When enabled, :hg:`clone` may download and apply a server-advertised
> +    bundle file from a URL instead of using the normal exchange mechanism.
> +
> +    This can likely result in faster and more reliable clones.
> +
> +    (default: True)
> +
>  ``clonebundlefallback``
>      Whether failure to apply an advertised "clone bundle" from a server
>      should result in fallback to a regular clone.
>
>      This is disabled by default because servers advertising "clone
>      bundles" often do so to reduce server load. If advertised bundles
>      start mass failing and clients automatically fall back to a regular
>      clone, this would add significant and unexpected load to the server
> diff --git a/tests/test-clonebundles.t b/tests/test-clonebundles.t
> --- a/tests/test-clonebundles.t
> +++ b/tests/test-clonebundles.t
> @@ -15,47 +15,26 @@ Set up a server
>    $ hg -q commit -A -m 'add foo'
>    $ touch bar
>    $ hg -q commit -A -m 'add bar'
>
>    $ hg serve -d -p $HGPORT --pid-file hg.pid --accesslog access.log
>    $ cat hg.pid >> $DAEMON_PIDS
>    $ cd ..
>
> -Feature disabled by default
> -(client should not request manifest)
> -
> -  $ hg clone -U http://localhost:$HGPORT feature-disabled
> -  requesting all changes
> -  adding changesets
> -  adding manifests
> -  adding file changes
> -  added 2 changesets with 2 changes to 2 files
> -
> -  $ cat server/access.log
> -  * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
> -  * - - [*] "GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D (glob)
> -  * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgarg-1:bundlecaps=HG20%2Cbundle2%3DHG20%250Achangegroup%253D01%252C02%252C03%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=1&common=0000000000000000000000000000000000000000&heads=aaff8d2ffbbf07a46dd1f05d8ae7877e3f56e2a2&listkeys=phase%2Cbookmarks (glob)
> -  * - - [*] "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=phases (glob)
> -
> -  $ cat >> $HGRCPATH << EOF
> -  > [experimental]
> -  > clonebundles = true
> -  > EOF
> -
>  Missing manifest should not result in server lookup
>
>    $ hg --verbose clone -U http://localhost:$HGPORT no-manifest
>    requesting all changes
>    adding changesets
>    adding manifests
>    adding file changes
>    added 2 changesets with 2 changes to 2 files
>
> -  $ tail -4 server/access.log
> +  $ cat server/access.log
>    * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
>    * - - [*] "GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D (glob)
>    * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgarg-1:bundlecaps=HG20%2Cbundle2%3DHG20%250Achangegroup%253D01%252C02%252C03%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=1&common=0000000000000000000000000000000000000000&heads=aaff8d2ffbbf07a46dd1f05d8ae7877e3f56e2a2&listkeys=phase%2Cbookmarks (glob)
>    * - - [*] "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=phases (glob)
>
>  Empty manifest file results in retrieval
>  (the extension only checks if the manifest file exists)
>
> @@ -70,38 +49,38 @@ Empty manifest file results in retrieval
>
>  Manifest file with invalid URL aborts
>
>    $ echo 'http://does.not.exist/bundle.hg' > server/.hg/clonebundles.manifest
>    $ hg clone http://localhost:$HGPORT 404-url
>    applying clone bundle from http://does.not.exist/bundle.hg
>    error fetching bundle: (.* not known|getaddrinfo failed) (re)
>    abort: error applying bundle
> -  (if this error persists, consider contacting the server operator or disable clone bundles via "--config experimental.clonebundles=false")
> +  (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
>    [255]
>
>  Server is not running aborts
>
>    $ echo "http://localhost:$HGPORT1/bundle.hg" > server/.hg/clonebundles.manifest
>    $ hg clone http://localhost:$HGPORT server-not-runner
>    applying clone bundle from http://localhost:$HGPORT1/bundle.hg
>    error fetching bundle: * refused* (glob)
>    abort: error applying bundle
> -  (if this error persists, consider contacting the server operator or disable clone bundles via "--config experimental.clonebundles=false")
> +  (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
>    [255]
>
>  Server returns 404
>
>    $ python $TESTDIR/dumbhttp.py -p $HGPORT1 --pid http.pid
>    $ cat http.pid >> $DAEMON_PIDS
>    $ hg clone http://localhost:$HGPORT running-404
>    applying clone bundle from http://localhost:$HGPORT1/bundle.hg
>    HTTP error fetching bundle: HTTP Error 404: File not found
>    abort: error applying bundle
> -  (if this error persists, consider contacting the server operator or disable clone bundles via "--config experimental.clonebundles=false")
> +  (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
>    [255]
>
>  We can override failure to fall back to regular clone
>
>    $ hg --config ui.clonebundlefallback=true clone -U http://localhost:$HGPORT 404-fallback
>    applying clone bundle from http://localhost:$HGPORT1/bundle.hg
>    HTTP error fetching bundle: HTTP Error 404: File not found
>    falling back to normal clone
> diff --git a/tests/test-help.t b/tests/test-help.t
> --- a/tests/test-help.t
> +++ b/tests/test-help.t
> @@ -247,17 +247,16 @@ Test extension help:
>        disabled extensions:
>
>         acl           hooks for controlling repository access
>         blackbox      log repository events to a blackbox for debugging
>         bugzilla      hooks for integrating with the Bugzilla bug tracker
>         censor        erase file content at a given revision
>         churn         command to display statistics about repository history
>         clonebundles  advertise pre-generated bundles to seed clones
> -                     (experimental)
>         color         colorize output from some commands
>         convert       import revisions from foreign VCS repositories into
>                       Mercurial
>         eol           automatically manage newlines in repository files
>         extdiff       command to allow external programs to compare revisions
>         factotum      http authentication with factotum
>         gpg           commands to sign and verify changesets
>         hgcia         hooks for integrating with the CIA.vc notification service
> @@ -1304,17 +1303,17 @@ Test keyword search help
>
>     bookmarks create a new bookmark or list existing bookmarks
>     clone     make a copy of an existing repository
>     paths     show aliases for remote repositories
>     update    update working directory (or switch revisions)
>
>    Extensions:
>
> -   clonebundles advertise pre-generated bundles to seed clones (experimental)
> +   clonebundles advertise pre-generated bundles to seed clones
>     prefixedname matched against word "clone"
>     relink       recreates hardlinks between repository clones
>
>    Extension Commands:
>
>     qclone clone main and patch repository at same time
>
>  Test unfound topic
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at selenic.com
> https://selenic.com/mailman/listinfo/mercurial-devel


More information about the Mercurial-devel mailing list