[PATCH 3 of 3 V2] exchange: make clone bundles non-experimental and enabled by default
Gregory Szorc
gregory.szorc at gmail.com
Fri Jan 8 13:02:21 CST 2016
# HG changeset patch
# User Gregory Szorc <gregory.szorc at gmail.com>
# Date 1452279484 28800
# Fri Jan 08 10:58:04 2016 -0800
# Node ID 31875bdd9fc69b4c68c2714ecc9c4a98a9e2275b
# Parent 074fc9366cbb65fd16d449bdc0f372317142b76a
exchange: make clone bundles non-experimental and enabled by default
The clone bundles feature was introduced in Mercurial 3.6 behind an
experimental and disabled by default flag. The feature has been enabled
on hg.mozilla.org for a few months and has served many terabytes of
clones. Users have been encouraged to use the feature and reception
has been very positive (mainly due to faster clones as a result of
connecting to a CDN). I have heard no feedback about changing the
feature other than inquiries about when it will be enabled by default.
So, I think the feature is ready to be enabled by default.
This patch renames experimental.clonebundles to ui.clonebundles,
documents the option, and enables it by default. References to the
experimental state of clone bundles have been removed. The remaining
config option docs in clonebundles.py have been removed because they
are redudant with `hg help config`.
There are some oddities with behavior of clone bundles. Because clones
with clone bundles are effectively 2 `hg pull` operations, there may be
2 transactions. This could result in hooks running twice. If the
subsequent pull is aborted, it could result in partial rollback and an
incomplete clone. This behavior is a bit wonky and should probably
be documented. If this patch is accepted, I'll send a follow-up to
document it. I don't think this behavior should prevent the feature
being enabled by default. Reworking the clone mechanism to support
interrupted or multi-part clones feels like a major new feature and
something that when implemented can change the hook and rollback
semantics of clone bundles. Besides, partial clone is better than
full rollback and hooks running on initial clone are likely rare, so I
think the impact is minimal.
diff --git a/hgext/clonebundles.py b/hgext/clonebundles.py
--- a/hgext/clonebundles.py
+++ b/hgext/clonebundles.py
@@ -1,12 +1,12 @@
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
-"""advertise pre-generated bundles to seed clones (experimental)
+"""advertise pre-generated bundles to seed clones
"clonebundles" is a server-side extension used to advertise the existence
of pre-generated, externally hosted bundle files to clients that are
cloning so that cloning can be faster, more reliable, and require less
resources on the server.
Cloning can be a CPU and I/O intensive operation on servers. Traditionally,
the server, in response to a client's request to clone, dynamically generates
@@ -155,26 +155,16 @@ Because there is no automatic Mercurial
bundle hosting service, it is important for server operators to view the bundle
hosting service as an extension of the Mercurial server in terms of
availability and service level agreements: if the bundle hosting service goes
down, so does the ability for clients to clone. Note: clients will see a
message informing them how to bypass the clone bundles facility when a failure
occurs. So server operators should prepare for some people to follow these
instructions when a failure occurs, thus driving more load to the original
Mercurial server when the bundle hosting service fails.
-
-The following config options influence the behavior of the clone bundles
-feature:
-
-ui.clonebundlefallback
- Whether to automatically fall back to a traditional clone in case of
- clone bundles failure. Defaults to false for reasons described above.
-
-experimental.clonebundles
- Whether the clone bundles feature is enabled on clients. Defaults to true.
"""
from mercurial import (
extensions,
wireproto,
)
testedwith = 'internal'
diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1670,17 +1670,17 @@ def unbundle(repo, cg, heads, source, ur
return r
def _maybeapplyclonebundle(pullop):
"""Apply a clone bundle from a remote, if possible."""
repo = pullop.repo
remote = pullop.remote
- if not repo.ui.configbool('experimental', 'clonebundles', False):
+ if not repo.ui.configbool('ui', 'clonebundles', True):
return
# Only run if local repo is empty.
if len(repo):
return
if pullop.heads:
return
@@ -1726,17 +1726,17 @@ def _maybeapplyclonebundle(pullop):
# clone load to be offloaded.
elif repo.ui.configbool('ui', 'clonebundlefallback', False):
repo.ui.warn(_('falling back to normal clone\n'))
else:
raise error.Abort(_('error applying bundle'),
hint=_('if this error persists, consider contacting '
'the server operator or disable clone '
'bundles via '
- '"--config experimental.clonebundles=false"'))
+ '"--config ui.clonebundles=false"'))
def parseclonebundlesmanifest(repo, s):
"""Parses the raw text of a clone bundles manifest.
Returns a list of dicts. The dicts have a ``URL`` key corresponding
to the URL and other keys are the attributes for the entry.
"""
m = []
diff --git a/mercurial/help/config.txt b/mercurial/help/config.txt
--- a/mercurial/help/config.txt
+++ b/mercurial/help/config.txt
@@ -1482,16 +1482,26 @@ User interface controls.
``askusername``
Whether to prompt for a username when committing. If True, and
neither ``$HGUSER`` nor ``$EMAIL`` has been specified, then the user will
be prompted to enter a username. If no username is entered, the
default ``USER at HOST`` is used instead.
(default: False)
+``clonebundles``
+ Whether the "clone bundles" feature is enabled.
+
+ When enabled, :hg:`clone` may download and apply a server-advertised
+ bundle file from a URL instead of using the normal exchange mechanism.
+
+ This can likely result in faster and more reliable clones.
+
+ (default: True)
+
``clonebundlefallback``
Whether failure to apply an advertised "clone bundle" from a server
should result in fallback to a regular clone.
This is disabled by default because servers advertising "clone
bundles" often do so to reduce server load. If advertised bundles
start mass failing and clients automatically fall back to a regular
clone, this would add significant and unexpected load to the server
diff --git a/tests/test-clonebundles.t b/tests/test-clonebundles.t
--- a/tests/test-clonebundles.t
+++ b/tests/test-clonebundles.t
@@ -15,47 +15,26 @@ Set up a server
$ hg -q commit -A -m 'add foo'
$ touch bar
$ hg -q commit -A -m 'add bar'
$ hg serve -d -p $HGPORT --pid-file hg.pid --accesslog access.log
$ cat hg.pid >> $DAEMON_PIDS
$ cd ..
-Feature disabled by default
-(client should not request manifest)
-
- $ hg clone -U http://localhost:$HGPORT feature-disabled
- requesting all changes
- adding changesets
- adding manifests
- adding file changes
- added 2 changesets with 2 changes to 2 files
-
- $ cat server/access.log
- * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
- * - - [*] "GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D (glob)
- * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgarg-1:bundlecaps=HG20%2Cbundle2%3DHG20%250Achangegroup%253D01%252C02%252C03%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=1&common=0000000000000000000000000000000000000000&heads=aaff8d2ffbbf07a46dd1f05d8ae7877e3f56e2a2&listkeys=phase%2Cbookmarks (glob)
- * - - [*] "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=phases (glob)
-
- $ cat >> $HGRCPATH << EOF
- > [experimental]
- > clonebundles = true
- > EOF
-
Missing manifest should not result in server lookup
$ hg --verbose clone -U http://localhost:$HGPORT no-manifest
requesting all changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 2 changes to 2 files
- $ tail -4 server/access.log
+ $ cat server/access.log
* - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
* - - [*] "GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D (glob)
* - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgarg-1:bundlecaps=HG20%2Cbundle2%3DHG20%250Achangegroup%253D01%252C02%252C03%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=1&common=0000000000000000000000000000000000000000&heads=aaff8d2ffbbf07a46dd1f05d8ae7877e3f56e2a2&listkeys=phase%2Cbookmarks (glob)
* - - [*] "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=phases (glob)
Empty manifest file results in retrieval
(the extension only checks if the manifest file exists)
@@ -70,38 +49,38 @@ Empty manifest file results in retrieval
Manifest file with invalid URL aborts
$ echo 'http://does.not.exist/bundle.hg' > server/.hg/clonebundles.manifest
$ hg clone http://localhost:$HGPORT 404-url
applying clone bundle from http://does.not.exist/bundle.hg
error fetching bundle: (.* not known|getaddrinfo failed) (re)
abort: error applying bundle
- (if this error persists, consider contacting the server operator or disable clone bundles via "--config experimental.clonebundles=false")
+ (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
[255]
Server is not running aborts
$ echo "http://localhost:$HGPORT1/bundle.hg" > server/.hg/clonebundles.manifest
$ hg clone http://localhost:$HGPORT server-not-runner
applying clone bundle from http://localhost:$HGPORT1/bundle.hg
error fetching bundle: * refused* (glob)
abort: error applying bundle
- (if this error persists, consider contacting the server operator or disable clone bundles via "--config experimental.clonebundles=false")
+ (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
[255]
Server returns 404
$ python $TESTDIR/dumbhttp.py -p $HGPORT1 --pid http.pid
$ cat http.pid >> $DAEMON_PIDS
$ hg clone http://localhost:$HGPORT running-404
applying clone bundle from http://localhost:$HGPORT1/bundle.hg
HTTP error fetching bundle: HTTP Error 404: File not found
abort: error applying bundle
- (if this error persists, consider contacting the server operator or disable clone bundles via "--config experimental.clonebundles=false")
+ (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
[255]
We can override failure to fall back to regular clone
$ hg --config ui.clonebundlefallback=true clone -U http://localhost:$HGPORT 404-fallback
applying clone bundle from http://localhost:$HGPORT1/bundle.hg
HTTP error fetching bundle: HTTP Error 404: File not found
falling back to normal clone
diff --git a/tests/test-help.t b/tests/test-help.t
--- a/tests/test-help.t
+++ b/tests/test-help.t
@@ -247,17 +247,16 @@ Test extension help:
disabled extensions:
acl hooks for controlling repository access
blackbox log repository events to a blackbox for debugging
bugzilla hooks for integrating with the Bugzilla bug tracker
censor erase file content at a given revision
churn command to display statistics about repository history
clonebundles advertise pre-generated bundles to seed clones
- (experimental)
color colorize output from some commands
convert import revisions from foreign VCS repositories into
Mercurial
eol automatically manage newlines in repository files
extdiff command to allow external programs to compare revisions
factotum http authentication with factotum
gpg commands to sign and verify changesets
hgcia hooks for integrating with the CIA.vc notification service
@@ -1304,17 +1303,17 @@ Test keyword search help
bookmarks create a new bookmark or list existing bookmarks
clone make a copy of an existing repository
paths show aliases for remote repositories
update update working directory (or switch revisions)
Extensions:
- clonebundles advertise pre-generated bundles to seed clones (experimental)
+ clonebundles advertise pre-generated bundles to seed clones
prefixedname matched against word "clone"
relink recreates hardlinks between repository clones
Extension Commands:
qclone clone main and patch repository at same time
Test unfound topic
More information about the Mercurial-devel
mailing list