Differences between revisions 18 and 19
Revision 18 as of 2015-04-19 02:50:34
Size: 6096
Comment:
Revision 19 as of 2015-04-19 04:29:30
Size: 6212
Comment:
Deletions are marked like this. Additions are marked like this.
Line 49: Line 49:
The following are best-of-5 timings on the Mozilla repo converted to GeneralDelta and manifestv2. ~28k recent revisions were rewritten in order to not give a too unfair advantage to tree manifests it would get due to short delta chains if only a few revisions had been rewritten. We will try to keep these numbers up to date as we work on tree manifests. The following are best-of-5 timings on the Mozilla repo converted to GeneralDelta. ~28k recent revisions were rewritten in order to not give a too unfair advantage to tree manifests it would get due to short delta chains if only a few revisions had been rewritten. We will try to keep these numbers up to date as we work on tree manifests.
Line 57: Line 57:
|| hg status --rev .~100 --rev . -C || 0.404 || 52.418 || 1.665 || 96.6% of "v2, flat" case spent in _slowreaddelta, 64.9% of "v2, tree" case spent in _adjustlinkrev || || hg status --rev .~100 --rev . -C || 0.404 || 52.418 || 1.665 || 64.9% of "v2, tree" case and 97.8% of "v2, flat" case spent in _adjustlinkrev. Almost all spent in _slowreaddelta in the latter case. The "v1, flat" case probably did not need linkrev adjustment due to how the repos were created. ||

Mercurial manifest sharding

Problem statement: imagine you have 1m to 1b files.

Individual manifest RAM overhead is a problem somewhere in this range.

Checkout: we don't want to materialize the working copy on the machine, and we don't want the whole manifest on the local machine.

Limitations of large and linear manifests:

  • manifest too large for RAM
  • manifest resolution too much CPU (long delta chains)
  • committing is slow because entire manifest has to be hashed
  • impossible for narrow clone to leave out part of manifest as all is needed to calculate new hash
  • diffing two revisions involves traversing entire subdirectories even if identical

A sample repository with 1M files is hosted on Google Drive (441MB).

1. Proposed Solution

Every directory will have its own manifest revlog. Each directory would thus have its own nodeid. This addresses the issues above:

  • entire manifest (including sub-manifests) usually does not need to be loaded
  • delta chains will be shorter since (sub-)manifests are smaller
  • since directories have nodeids, directories not touched by the commit don't need to hashed
  • narrow clone will be able to skip unwanted directories and files and will still be able to calculate a new hash
  • diffing becomes faster since directory nodeids are stored and entire directories can be skipped

Some further benefits:

  • space savings by only storing the filename in the manifest (but also see Costs below)
  • 'hg log path/to/dir' can be made faster by walking the directory's revlog.

Costs:

  • more revlogs need to be stored, using more space (+7% number of revlogs for the Mozilla repo, +20%/80k for a Facebook repo)
  • more revlogs means more disk seeks
  • when a single file many levels down in a directory is changed, many revlogs need to be written

2. Current Plan

martinvonz to follow these steps:

  1. Parse manifests into a tree data structure when config option set
  2. When the config option is set, store one manifest revlog per directory
  3. Compare performance on large repository (e.g. Firefox) and improve performance
  4. Hack together some experimental narrow functionality in order to see how things work with push/pull over the network

3. Performance

The following are best-of-5 timings on the Mozilla repo converted to GeneralDelta. ~28k recent revisions were rewritten in order to not give a too unfair advantage to tree manifests it would get due to short delta chains if only a few revisions had been rewritten. We will try to keep these numbers up to date as we work on tree manifests.

Command

v1, flat

v2, flat

v2, tree

Comments

hg files -r .

0.791

1.174s

2.318s

Particularly parsing is not yet optimized, but tree still expected to be slower than flat

hg files -r . python/

0.379

0.743s

0.149s

~800 files out of 115k files

hg diff --change .

0.624

1.456

0.618s

A ~7k-line diff

hg status --rev .~1000 --rev .

0.411

1.236

1.185

~43k differing files

hg status --rev .~10000 --rev .

1.080

1.634

2.362

~8k differing files

hg status --rev .~100 --rev . -C

0.404

52.418

1.665

64.9% of "v2, tree" case and 97.8% of "v2, flat" case spent in _adjustlinkrev. Almost all spent in _slowreaddelta in the latter case. The "v1, flat" case probably did not need linkrev adjustment due to how the repos were created.

hg rebase --keep -d new-tip~10 -r new-tip~8

-

4.573s

3.653s

~60% spent in dirstate.status

hg log --limit 10 -p python/

3.437

10.999s

0.526s

4. Alternatives considered

4.1. Sub-manifests at custom positions in tree

A user splits a shard out using a command like hg debugmarkshard foo/bar/baz, which is then stored as a sub-manifest in a different revlog.

Challenges:

  • someone has to manage the shards
  • merging shard boundaries has to happen

Objections:

  • shard boundaries get encoded in the revision history
    • durin42 and durham find this displeasing
    • it also means users can't have a manifest that's exactly as narrow as their client

4.2. Tree-state hash, but flat manifest

Make manifest hash something clients with only a partial checkout can do is to do a per-directory hash that bubbles up, and store entries for those directory nodes in their parent with a d in the flags entry. We considered using a hash of filename and hash mod == 0 do a shard, but decided that was probably going to lead to lots of churn, and also bakes the sharding scheme into the manifest hash (which might be suboptimal).

Will require client support to do pull of sharded manifests - that's the second step.

Challenges:

  • means actually breaking out a shard is expensive, as you have to split/join for network traffic
  • pushing means the server has to produce a matching-spec narrow manifest to apply the delta
    • durham proposes we could store the number of bytes the client had elided in the manifest, which would allow us to produce the same delta as though we were operating on the full manifest, and apply full-manifest deltas even when they contained bits we didn't care about.
  • need out-of-manifest management of some sort?
  • doesn't help reduce length of delta chains
  • doesn't help with working with manifests too large for RAM

5. Related: sparse checkouts

Currently hg sparse --include mobile/

doesn't matter if the repo has other stuff, you only get the mobile directory.

hg sparse --enable-profile mobile

profiles live in repo. .hgsparse

proposal: have team specific .hgsparse files in directories. Allows changes without contention. (hg sparse --enable-profile mobile[/.hgsparse])

future magic: hg clone --sparse mobile (to avoid initial full clone)

merges get a little complicated using regexps matching now. proposed to use directories for includes, but allow regex/glob for exclude (to allow not writing certain types of files, like photoshop files)

6. narrow changelog

See NarrowClonePlan

TreeManifestPlan (last edited 2016-10-06 16:21:31 by MartinVonZweigbergk)