Mercurial manifest sharding

Dealing with big repos.

1. Introduction

Problem statement: imagine you have 1M to 1B files.

Individual manifest RAM overhead is a problem somewhere in this range.

Checkout: we don't want to materialize the working copy on the machine, and we don't want the whole manifest on the local machine.

Limitations of large and linear manifests:

A sample repository with 1M files is hosted on Google Drive (441MB).

2. Proposed Solution

Every directory will have its own manifest revlog. Each directory would thus have its own nodeid. This addresses the issues above:

Some further benefits:

Costs:

3. Current Plan

martinvonz to follow these steps:

  1. Parse manifests into a tree data structure when config option set (Done)
  2. When the config option is set, store one manifest revlog per directory (Done)
  3. Compare performance on large repository (e.g. Firefox) and improve performance (Done)
  4. Hack together some experimental narrow functionality in order to see how things work with push/pull over the network (Done)

4. Performance

The following are best-of-5 timings on the Mozilla repo converted to GeneralDelta. ~88k recent revisions were rewritten in order to not give a too unfair advantage to tree manifests it would get due to short delta chains if only a few revisions had been rewritten.

Command

v1, flat

v1, tree

Comments

hg files -r .

0.79

2.52

Particularly parsing is not yet optimized, but tree still expected to be slower than flat

hg files -r . python/

0.46

0.17

1075 files out of 119k files

hg diff --change .

0.91

0.61

A ~7k-line diff

hg status --rev .~100 --rev .

0.65

0.74

1585 differing files

hg status --rev .~10000 --rev .

0.97

2.62

~43k differing files

hg status --rev .~10000 --rev . -C python

0.93

0.41

632 differing files

hg rebase --keep -d x~10 -r x~8::x~4

9.89

7.36

hg log --limit 10 -p python/

3.87

0.43

5. Alternatives considered

5.1. Sub-manifests at custom positions in tree

A user splits a shard out using a command like hg debugmarkshard foo/bar/baz, which is then stored as a sub-manifest in a different revlog.

Challenges:

Objections:

5.2. Tree-state hash, but flat manifest

Make manifest hashing something clients with only a partial checkout can do by doing a per-directory hash that bubbles up, and store entries for those directory nodes in their parent with a d in the flags entry. We considered using a hash of filename and hash mod == 0 do a shard, but decided that was probably going to lead to lots of churn, and also bakes the sharding scheme into the manifest hash (which might be suboptimal).

Will require client support to do pull of sharded manifests - that's the second step.

Challenges:

6. Related: sparse checkouts

Currently hg sparse --include mobile/

doesn't matter if the repo has other stuff, you only get the mobile directory.

hg sparse --enable-profile mobile

profiles live in repo. .hgsparse

proposal: have team specific .hgsparse files in directories. Allows changes without contention. (hg sparse --enable-profile mobile[/.hgsparse])

future magic: hg clone --sparse mobile (to avoid initial full clone)

merges get a little complicated using regexps matching now. proposed to use directories for includes, but allow regex/glob for exclude (to allow not writing certain types of files, like photoshop files)

7. narrow changelog

See NarrowClonePlan


CategoryDeveloper CategoryNewFeatures

TreeManifestPlan (last edited 2016-10-06 16:21:31 by MartinVonZweigbergk)