clonebundles

The clonebundles extension is a server-side extension that allows servers to advertise URLs of pre-generated bundles to be used during clone operations. When clients clone the repository, instead of the server dynamically creating a bundle and sending it to the client (this can be CPU, I/O, memory, and network intensive), the client is effectively redirected to another URL. This can drastically reduce server load when processing clone requests.

1. Status

This extension is distributed with Mercurial and is in use in production environments.

Mozilla uses this extension on https://hg.mozilla.org/ to offload clones of the 1+ GB Firefox repository to a CDN and Amazon S3, resulting in 10+ TB/day traffic reduction from the Mercurial server and a 90+% reduction in server CPU load. Read more at https://gregoryszorc.com/blog/2018/07/27/benefits-of-clone-offload-on-version-control-hosting/.

2. Overview

When a client performs a hg clone, the remote server will normally dynamically create a bundle of repository content. Essentially all repository data is scanned, compressed, and sent to the client. This can be CPU, I/O, memory, and network intensive, especially on large repositories. If 2 clients perform hg clone, the server performs the same bundling work simultaneously, resulting in double the server load. The scaling problem should be obvious.

Clonebundles is effectively a cache for clone data. The server operator pre-generates a bundle file and uploads it somewhere. When a client performs a hg clone, the server tells the client to fetch that pre-generated file. The client does so then goes back to the server and fetches all data produced since that bundle was created. Instead of the server processing and sending all repository data for each clone request, it only has to process and send the data newer than the advertised bundle file. In practice, this effectively makes CPU load for processing clones negligible.

3. Configuration

Configure your .hgrc to enable the extension by adding following lines:

[extensions]
clonebundles =

A .hg/clonebundles.manifest file will need to be installed in each repository to advertise URLs of bundles. The format of this file is documented in hg help -e clonebundles.

4. See also


CategoryBundledExtension

ClonebundlesExtension (last edited 2019-05-31 02:53:51 by GregorySzorc)