Differences between revisions 6 and 7
Revision 6 as of 2014-11-13 19:50:42
Size: 4844
Editor: AugieFackler
Comment:
Revision 7 as of 2015-02-10 22:22:37
Size: 5005
Comment: Added more limitations to list and removed "narrow clone" from scope of this document
Deletions are marked like this. Additions are marked like this.
Line 8: Line 8:
Limitations of large manifests/repos: Limitations of large and linear manifests:
Line 11: Line 11:
 * checkout too large for local disk
 * clone size too large for local disk
 * manifest resolution too much CPU
 * 100k+ files on HFS+ has bad perf
 * manifest resolution too much CPU (long delta chains)
 * committing is slow because entire manifest has to be hashed
 * impossible for narrow clone to leave out part of manifest as all is needed to calculate new hash
 * diffing two revisions involves traversing entire subdirectories even if identical

Mercurial manifest sharding

Problem statement: imagine you have 1m to 1b files.

Individual manifest RAM overhead is a problem somewhere in this range.

Checkout: we don't want to materialize the working copy on the machine, and we don't want the whole manifest on the local machine.

Limitations of large and linear manifests:

  • manifest too large for RAM
  • manifest resolution too much CPU (long delta chains)
  • committing is slow because entire manifest has to be hashed
  • impossible for narrow clone to leave out part of manifest as all is needed to calculate new hash
  • diffing two revisions involves traversing entire subdirectories even if identical

Two possible paths forward: explicit shard boundaries or doing a tree-state hash that can elide uninteresting-to-a-client subdirectories.

A sample repository with 1M files is hosted on Google Drive (441MB).

0.1. Tree-state hash

Current plan to make manifest hash something clients with only a partial checkout can do is to do a per-directory hash that bubbles up, and store entries for those directory nodes in their parent with a d in the flags entry. We considered using a hash of filename and hash mod == 0 do a shard, but decided that was probably going to lead to lots of churn, and also bakes the sharding scheme into the manifest hash (which might be suboptimal).

Will require client support to do pull of sharded manifests - that's the second step.

Challenges:

  • means actually breaking out a shard is expensive, as you have to split/join for network traffic
  • pushing means the server has to produce a matching-spec narrow manifest to apply the delta
    • durham proposes we could store the number of bytes the client had elided in the manifest, which would allow us to produce the same delta as though we were operating on the full manifest, and apply full-manifest deltas even when they contained bits we didn't care about.
  • need out-of-manifest management of some sort?

0.2. Explicit shards

A user splits a shard out using a command like hg debugmarkshard foo/bar/baz, which is then stored as a sub-manifest in a different revlog.

Challenges:

  • someone has to manage the shards
  • merging shard boundaries has to happen

Objections:

  • shard boundaries get encoded in the revision history
    • durin42 and durham find this displeasing
    • it also means users can't have a manifest that's exactly as narrow as their client

durin42 thinks that the normal case for a user with a giant repo is that their manifest will always be of a reasonable size since they have to fit all those files on disk.

1. Current Plan

durin42 to follow these steps:

  1. Make manifest parsing lazy (similar to parse_index2)
    • - this will make manifest lookups be O(log(manifest size)) and iteration of the manifest in lexicographic order be O(n) instead of requiring a sort.
  2. Add a manifest class that stores the directory nodes and uses the alternate (tree-state) hash for nodes
  3. Hack together some experimental narrow functionality in order to see how things work with push/pull over the network

This doesn't open the door to an actually-sharded manifest, but it may not matter.

2. Discussion (from titanpad)

Directory recursive hashes: We could compute the hash for submanifests by iterating recursively over the directories in the sub manifest content. This would produce a hash that is unique to the commit directory structure, and agnostic to how the manifest is sharded. (similar to git tree hash calculations). Pros:

  • allows changing the manifest format in the future without changing the hashes
  • allows delivering customly sharded manifests to users on demand

Cons:

  • more expensive hash algorithm
  • will require a new manifest version flag (since it won't be backwards comptabile)

3. Related: sparse checkouts

Currently hg sparse --include mobile/

doesn't matter if the repo has other stuff, you only get the mobile directory.

hg sparse --enable-profile mobile

profiles live in repo. .hgsparse

proposal: have team specific .hgsparse files in directories. Allows changes without contention. (hg sparse --enable-profile mobile[/.hgsparse])

future magic: hg clone --sparse mobile (to avoid initial full clone)

merges get a little complicated using regexps matching now. proposed to use directories for includes, but allow regex/glob for exclude (to allow not writing certain types of files, like photoshop files)

4. narrow changelog (this was part of the discussion at the 3.2Sprint, but should probably be its own page)

ellipsis nodes listing heads and roots

running log and bisect inside ellipsis is difficult. change default so hg log only considers downloaded changesets by default?

DAG can have dangling links and won't explode when we hit them?

Should we have remote fallback for hg log? durin42 thinks so.

TreeManifestPlan (last edited 2016-10-06 16:21:31 by MartinVonZweigbergk)