[PATCH 1 of 5] sparse-revlog: new requirement enabled with format.sparse-revlog

[Gregory Szorc]

On Mon, Jul 16, 2018 at 11:50 AM, Boris Feld <boris.feld at octobus.net> wrote:

> # HG changeset patch
> # User Paul Morelle <paul.morelle at octobus.net>
> # Date 1528143798 -7200
> #      Mon Jun 04 22:23:18 2018 +0200
> # Node ID 6ed641517622f53dd51193c3ae3930c66fc18078
> # Parent  cc76692f401d0063cd5dfaf46f0fca1563a85fae
> # EXP-Topic write-for-sparse-read
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #              hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
> 6ed641517622
> sparse-revlog: new requirement enabled with format.sparse-revlog
>

Queued this series, thanks.

I did flip the order of parts 1 and 2 when landing so all the sparse
commits are grouped.

I'm reasonably convinced this code won't have a negative impact on repos
that haven't opted into the feature. So it feels safe to land just before
feature freeze.


>
> The meaning of the new 'sparse-revlog' requirement is that the revlogs are
> allowed to contain wider delta chains with larger holes between the
> interesting
> chunks. These sparse delta chains should be read in several chunks to
> avoid a
> potential explosion of memory usage.
>
> Former version won't know how to read a delta chain in several chunks. They
> would keep reading them in a single read, and therefore would be subject
> to the
> potential memory explosion. Hence this new requirement: only versions
> having
> support of sparse-revlog reading should be allowed to read such a revlog.
>
> Implementation of this new algorithm and tools to enable or disable the
> requirement will follow in the next changesets.
>
> diff --git a/mercurial/configitems.py b/mercurial/configitems.py
> --- a/mercurial/configitems.py
> +++ b/mercurial/configitems.py
> @@ -652,6 +652,9 @@ coreconfigitem('format', 'maxchainlen',
>  coreconfigitem('format', 'obsstore-version',
>      default=None,
>  )
> +coreconfigitem('format', 'sparse-revlog',
> +    default=False,
> +)
>  coreconfigitem('format', 'usefncache',
>      default=True,
>  )
> diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
> --- a/mercurial/localrepo.py
> +++ b/mercurial/localrepo.py
> @@ -354,6 +354,15 @@ class locallegacypeer(localpeer):
>  # clients.
>  REVLOGV2_REQUIREMENT = 'exp-revlogv2.0'
>
> +# A repository with the sparserevlog feature will have delta chains that
> +# can spread over a larger span. Sparse reading cuts these large spans
> into
> +# pieces, so that each piece isn't too big.
> +# Without the sparserevlog capability, reading from the repository could
> use
> +# huge amounts of memory, because the whole span would be read at once,
> +# including all the intermediate revisions that aren't pertinent for the
> chain.
> +# This is why once a repository has enabled sparse-read, it becomes
> required.
> +SPARSEREVLOG_REQUIREMENT = 'sparserevlog'
> +
>  # Functions receiving (ui, features) that extensions can register to
> impact
>  # the ability to load repositories with custom requirements. Only
>  # functions defined in loaded extensions are called.
> @@ -376,6 +385,7 @@ class localrepository(object):
>          'generaldelta',
>          'treemanifest',
>          REVLOGV2_REQUIREMENT,
> +        SPARSEREVLOG_REQUIREMENT,
>      }
>      _basesupported = supportedformats | {
>          'store',
> @@ -678,6 +688,8 @@ class localrepository(object):
>          self.svfs.options['with-sparse-read'] = withsparseread
>          self.svfs.options['sparse-read-density-threshold'] =
> srdensitythres
>          self.svfs.options['sparse-read-min-gap-size'] = srmingapsize
> +        sparserevlog = SPARSEREVLOG_REQUIREMENT in self.requirements
> +        self.svfs.options['sparse-revlog'] = sparserevlog
>
>          for r in self.requirements:
>              if r.startswith('exp-compression-'):
> @@ -2370,6 +2382,9 @@ def newreporequirements(repo):
>          requirements.add('generaldelta')
>      if ui.configbool('experimental', 'treemanifest'):
>          requirements.add('treemanifest')
> +    # experimental config: format.sparse-revlog
> +    if ui.configbool('format', 'sparse-revlog'):
> +        requirements.add(SPARSEREVLOG_REQUIREMENT)
>
>      revlogv2 = ui.config('experimental', 'revlogv2')
>      if revlogv2 == 'enable-unstable-format-and-corrupt-my-data':
> diff --git a/mercurial/revlog.py b/mercurial/revlog.py
> --- a/mercurial/revlog.py
> +++ b/mercurial/revlog.py
> @@ -895,6 +895,7 @@ class revlog(object):
>          self._compengine = 'zlib'
>          self._maxdeltachainspan = -1
>          self._withsparseread = False
> +        self._sparserevlog = False
>          self._srdensitythreshold = 0.50
>          self._srmingapsize = 262144
>
> @@ -923,7 +924,10 @@ class revlog(object):
>                  self._maxdeltachainspan = opts['maxdeltachainspan']
>              if mmaplargeindex and 'mmapindexthreshold' in opts:
>                  mmapindexthreshold = opts['mmapindexthreshold']
> -            self._withsparseread = bool(opts.get('with-sparse-read',
> False))
> +            self._sparserevlog = bool(opts.get('sparse-revlog', False))
> +            withsparseread = bool(opts.get('with-sparse-read', False))
> +            # sparse-revlog forces sparse-read
> +            self._withsparseread = self._sparserevlog or withsparseread
>              if 'sparse-read-density-threshold' in opts:
>                  self._srdensitythreshold = opts['sparse-read-density-
> threshold']
>              if 'sparse-read-min-gap-size' in opts:
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20180716/5ea5c7e0/attachment.html>