[PATCH 4 of 4 V2] clone: add a server-side option to disable full getbundles (pull-based clones)

[Augie Fackler]

> On May 12, 2017, at 23:08, Gregory Szorc <gregory.szorc at gmail.com> wrote:
> 
> On Fri, May 12, 2017 at 7:22 PM, Augie Fackler <raf at durin42.com> wrote:
> On Thu, May 11, 2017 at 10:51:23AM -0700, Siddharth Agarwal wrote:
> > # HG changeset patch
> > # User Siddharth Agarwal <sid0 at fb.com>
> > # Date 1494525005 25200
> > #      Thu May 11 10:50:05 2017 -0700
> > # Node ID fa570690257581e3197f8730b6522e80d58a3f45
> > # Parent  052bd5cfe3769b10c64a4a39d9734a2740d44e16
> > clone: add a server-side option to disable full getbundles (pull-based clones)
> 
> queued, thanks
> 
> very nice - I can think of some repositories that should enable
> clonebundles and this new rejection mechanism to save tons of server
> load. :)
> 
> FWIW, it is common for >95% of hg.mozilla.org's served bytes to be handled from clone bundles via S3/CDN. Our daily record saw 42,760,071,088,706 of 43,134,314,834,541 (99.13%) of bytes served from S3. Most of this load is Firefox's CI cloning the Firefox repositories. We make heavy use of volatile "spot" instances rather than permanent infrastructure, so there's a lot of cloning go on, even with aggressive reuse of clones on clients.
> 
> If we didn't have clone bundles, we'd need to roll our own caching layer and/or pay a heavy cost for extra server infrastructure. Ironically, clone bundles has offloaded so much CPU from servers that we can afford to take computational hits on the server, such as allowing bundle1 clients to clone/pull from generaldelta repos and leaving full bundle clones enabled. But I'm still glad we have config knobs to lock out legacy clients at the first sign of scaling trouble.

I was actually thinking of the Adium "full history" repo which is a few gigs and has had pull/clone disabled since it came into existence. But yes, your repos too. :)