[PATCH 3 of 3 V2] exchange: make clone bundles non-experimental and enabled by default
Augie Fackler
raf at durin42.com
Tue Jan 12 09:12:30 CST 2016
On Fri, Jan 08, 2016 at 11:02:21AM -0800, Gregory Szorc wrote:
> # HG changeset patch
> # User Gregory Szorc <gregory.szorc at gmail.com>
> # Date 1452279484 28800
> # Fri Jan 08 10:58:04 2016 -0800
> # Node ID 31875bdd9fc69b4c68c2714ecc9c4a98a9e2275b
> # Parent 074fc9366cbb65fd16d449bdc0f372317142b76a
> exchange: make clone bundles non-experimental and enabled by default
Queued! Thanks for taking my hacky proof of concept and turning it in
to something useful.
>
> The clone bundles feature was introduced in Mercurial 3.6 behind an
> experimental and disabled by default flag. The feature has been enabled
> on hg.mozilla.org for a few months and has served many terabytes of
> clones. Users have been encouraged to use the feature and reception
> has been very positive (mainly due to faster clones as a result of
> connecting to a CDN). I have heard no feedback about changing the
> feature other than inquiries about when it will be enabled by default.
> So, I think the feature is ready to be enabled by default.
>
> This patch renames experimental.clonebundles to ui.clonebundles,
> documents the option, and enables it by default. References to the
> experimental state of clone bundles have been removed. The remaining
> config option docs in clonebundles.py have been removed because they
> are redudant with `hg help config`.
>
> There are some oddities with behavior of clone bundles. Because clones
> with clone bundles are effectively 2 `hg pull` operations, there may be
> 2 transactions. This could result in hooks running twice. If the
> subsequent pull is aborted, it could result in partial rollback and an
> incomplete clone. This behavior is a bit wonky and should probably
> be documented. If this patch is accepted, I'll send a follow-up to
> document it. I don't think this behavior should prevent the feature
> being enabled by default. Reworking the clone mechanism to support
> interrupted or multi-part clones feels like a major new feature and
> something that when implemented can change the hook and rollback
> semantics of clone bundles. Besides, partial clone is better than
> full rollback and hooks running on initial clone are likely rare, so I
> think the impact is minimal.
>
> diff --git a/hgext/clonebundles.py b/hgext/clonebundles.py
> --- a/hgext/clonebundles.py
> +++ b/hgext/clonebundles.py
> @@ -1,12 +1,12 @@
> # This software may be used and distributed according to the terms of the
> # GNU General Public License version 2 or any later version.
>
> -"""advertise pre-generated bundles to seed clones (experimental)
> +"""advertise pre-generated bundles to seed clones
>
> "clonebundles" is a server-side extension used to advertise the existence
> of pre-generated, externally hosted bundle files to clients that are
> cloning so that cloning can be faster, more reliable, and require less
> resources on the server.
>
> Cloning can be a CPU and I/O intensive operation on servers. Traditionally,
> the server, in response to a client's request to clone, dynamically generates
> @@ -155,26 +155,16 @@ Because there is no automatic Mercurial
> bundle hosting service, it is important for server operators to view the bundle
> hosting service as an extension of the Mercurial server in terms of
> availability and service level agreements: if the bundle hosting service goes
> down, so does the ability for clients to clone. Note: clients will see a
> message informing them how to bypass the clone bundles facility when a failure
> occurs. So server operators should prepare for some people to follow these
> instructions when a failure occurs, thus driving more load to the original
> Mercurial server when the bundle hosting service fails.
> -
> -The following config options influence the behavior of the clone bundles
> -feature:
> -
> -ui.clonebundlefallback
> - Whether to automatically fall back to a traditional clone in case of
> - clone bundles failure. Defaults to false for reasons described above.
> -
> -experimental.clonebundles
> - Whether the clone bundles feature is enabled on clients. Defaults to true.
> """
>
> from mercurial import (
> extensions,
> wireproto,
> )
>
> testedwith = 'internal'
> diff --git a/mercurial/exchange.py b/mercurial/exchange.py
> --- a/mercurial/exchange.py
> +++ b/mercurial/exchange.py
> @@ -1670,17 +1670,17 @@ def unbundle(repo, cg, heads, source, ur
> return r
>
> def _maybeapplyclonebundle(pullop):
> """Apply a clone bundle from a remote, if possible."""
>
> repo = pullop.repo
> remote = pullop.remote
>
> - if not repo.ui.configbool('experimental', 'clonebundles', False):
> + if not repo.ui.configbool('ui', 'clonebundles', True):
> return
>
> # Only run if local repo is empty.
> if len(repo):
> return
>
> if pullop.heads:
> return
> @@ -1726,17 +1726,17 @@ def _maybeapplyclonebundle(pullop):
> # clone load to be offloaded.
> elif repo.ui.configbool('ui', 'clonebundlefallback', False):
> repo.ui.warn(_('falling back to normal clone\n'))
> else:
> raise error.Abort(_('error applying bundle'),
> hint=_('if this error persists, consider contacting '
> 'the server operator or disable clone '
> 'bundles via '
> - '"--config experimental.clonebundles=false"'))
> + '"--config ui.clonebundles=false"'))
>
> def parseclonebundlesmanifest(repo, s):
> """Parses the raw text of a clone bundles manifest.
>
> Returns a list of dicts. The dicts have a ``URL`` key corresponding
> to the URL and other keys are the attributes for the entry.
> """
> m = []
> diff --git a/mercurial/help/config.txt b/mercurial/help/config.txt
> --- a/mercurial/help/config.txt
> +++ b/mercurial/help/config.txt
> @@ -1482,16 +1482,26 @@ User interface controls.
>
> ``askusername``
> Whether to prompt for a username when committing. If True, and
> neither ``$HGUSER`` nor ``$EMAIL`` has been specified, then the user will
> be prompted to enter a username. If no username is entered, the
> default ``USER at HOST`` is used instead.
> (default: False)
>
> +``clonebundles``
> + Whether the "clone bundles" feature is enabled.
> +
> + When enabled, :hg:`clone` may download and apply a server-advertised
> + bundle file from a URL instead of using the normal exchange mechanism.
> +
> + This can likely result in faster and more reliable clones.
> +
> + (default: True)
> +
> ``clonebundlefallback``
> Whether failure to apply an advertised "clone bundle" from a server
> should result in fallback to a regular clone.
>
> This is disabled by default because servers advertising "clone
> bundles" often do so to reduce server load. If advertised bundles
> start mass failing and clients automatically fall back to a regular
> clone, this would add significant and unexpected load to the server
> diff --git a/tests/test-clonebundles.t b/tests/test-clonebundles.t
> --- a/tests/test-clonebundles.t
> +++ b/tests/test-clonebundles.t
> @@ -15,47 +15,26 @@ Set up a server
> $ hg -q commit -A -m 'add foo'
> $ touch bar
> $ hg -q commit -A -m 'add bar'
>
> $ hg serve -d -p $HGPORT --pid-file hg.pid --accesslog access.log
> $ cat hg.pid >> $DAEMON_PIDS
> $ cd ..
>
> -Feature disabled by default
> -(client should not request manifest)
> -
> - $ hg clone -U http://localhost:$HGPORT feature-disabled
> - requesting all changes
> - adding changesets
> - adding manifests
> - adding file changes
> - added 2 changesets with 2 changes to 2 files
> -
> - $ cat server/access.log
> - * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
> - * - - [*] "GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D (glob)
> - * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgarg-1:bundlecaps=HG20%2Cbundle2%3DHG20%250Achangegroup%253D01%252C02%252C03%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=1&common=0000000000000000000000000000000000000000&heads=aaff8d2ffbbf07a46dd1f05d8ae7877e3f56e2a2&listkeys=phase%2Cbookmarks (glob)
> - * - - [*] "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=phases (glob)
> -
> - $ cat >> $HGRCPATH << EOF
> - > [experimental]
> - > clonebundles = true
> - > EOF
> -
> Missing manifest should not result in server lookup
>
> $ hg --verbose clone -U http://localhost:$HGPORT no-manifest
> requesting all changes
> adding changesets
> adding manifests
> adding file changes
> added 2 changesets with 2 changes to 2 files
>
> - $ tail -4 server/access.log
> + $ cat server/access.log
> * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
> * - - [*] "GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D (glob)
> * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgarg-1:bundlecaps=HG20%2Cbundle2%3DHG20%250Achangegroup%253D01%252C02%252C03%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=1&common=0000000000000000000000000000000000000000&heads=aaff8d2ffbbf07a46dd1f05d8ae7877e3f56e2a2&listkeys=phase%2Cbookmarks (glob)
> * - - [*] "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=phases (glob)
>
> Empty manifest file results in retrieval
> (the extension only checks if the manifest file exists)
>
> @@ -70,38 +49,38 @@ Empty manifest file results in retrieval
>
> Manifest file with invalid URL aborts
>
> $ echo 'http://does.not.exist/bundle.hg' > server/.hg/clonebundles.manifest
> $ hg clone http://localhost:$HGPORT 404-url
> applying clone bundle from http://does.not.exist/bundle.hg
> error fetching bundle: (.* not known|getaddrinfo failed) (re)
> abort: error applying bundle
> - (if this error persists, consider contacting the server operator or disable clone bundles via "--config experimental.clonebundles=false")
> + (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
> [255]
>
> Server is not running aborts
>
> $ echo "http://localhost:$HGPORT1/bundle.hg" > server/.hg/clonebundles.manifest
> $ hg clone http://localhost:$HGPORT server-not-runner
> applying clone bundle from http://localhost:$HGPORT1/bundle.hg
> error fetching bundle: * refused* (glob)
> abort: error applying bundle
> - (if this error persists, consider contacting the server operator or disable clone bundles via "--config experimental.clonebundles=false")
> + (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
> [255]
>
> Server returns 404
>
> $ python $TESTDIR/dumbhttp.py -p $HGPORT1 --pid http.pid
> $ cat http.pid >> $DAEMON_PIDS
> $ hg clone http://localhost:$HGPORT running-404
> applying clone bundle from http://localhost:$HGPORT1/bundle.hg
> HTTP error fetching bundle: HTTP Error 404: File not found
> abort: error applying bundle
> - (if this error persists, consider contacting the server operator or disable clone bundles via "--config experimental.clonebundles=false")
> + (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
> [255]
>
> We can override failure to fall back to regular clone
>
> $ hg --config ui.clonebundlefallback=true clone -U http://localhost:$HGPORT 404-fallback
> applying clone bundle from http://localhost:$HGPORT1/bundle.hg
> HTTP error fetching bundle: HTTP Error 404: File not found
> falling back to normal clone
> diff --git a/tests/test-help.t b/tests/test-help.t
> --- a/tests/test-help.t
> +++ b/tests/test-help.t
> @@ -247,17 +247,16 @@ Test extension help:
> disabled extensions:
>
> acl hooks for controlling repository access
> blackbox log repository events to a blackbox for debugging
> bugzilla hooks for integrating with the Bugzilla bug tracker
> censor erase file content at a given revision
> churn command to display statistics about repository history
> clonebundles advertise pre-generated bundles to seed clones
> - (experimental)
> color colorize output from some commands
> convert import revisions from foreign VCS repositories into
> Mercurial
> eol automatically manage newlines in repository files
> extdiff command to allow external programs to compare revisions
> factotum http authentication with factotum
> gpg commands to sign and verify changesets
> hgcia hooks for integrating with the CIA.vc notification service
> @@ -1304,17 +1303,17 @@ Test keyword search help
>
> bookmarks create a new bookmark or list existing bookmarks
> clone make a copy of an existing repository
> paths show aliases for remote repositories
> update update working directory (or switch revisions)
>
> Extensions:
>
> - clonebundles advertise pre-generated bundles to seed clones (experimental)
> + clonebundles advertise pre-generated bundles to seed clones
> prefixedname matched against word "clone"
> relink recreates hardlinks between repository clones
>
> Extension Commands:
>
> qclone clone main and patch repository at same time
>
> Test unfound topic
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at selenic.com
> https://selenic.com/mailman/listinfo/mercurial-devel
More information about the Mercurial-devel
mailing list