D2883: revlogstore: create and implement an interface for repo files storage

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  In order to better support partial clones, we will need to overhaul
  local repository storage. This will be a major effort, as many parts
  of the code assume things like the existence of revlogs for storing
  data.
  
  To help support alternate storage implementations, we will create
  interfaces for accessing storage. The idea is that consumers will
  all code to an interface and any new interface-conforming
  implementation can come along and be swapped in to provide new and
  novel storage mechanisms.
  
  This commit starts the process of defining those interfaces.
  
  We define an interface for accessing files data. It has a single
  method for resolving the fulltext of an iterable of inputs.
  
  The interface is specifically defined to allow out-of-order responses.
  It also provides a mechanism for declaring that files data is censored.
  We *may* also want a mechanism to declare LFS or largefiles data.
  But I'm not sure how that mechanism works or what the best way to
  handle that would be, if any.
  
  We introduce a new "revlogstore" module to hold the definitions of
  these interfaces that use our existing revlog-based storage
  mechanism.
  
  An attribute pointing to the "files store" has been added to
  localrepository.
  
  No consumers of the new interface have been added. The interface
  should still be considered highly experimental and details are
  expected to change.
  
  It was tempting to define the interface as one level higher than
  file storage - in such a way to facilitate accessing changeset
  and manifest data as well. However, I believe these 3 primitives -
  changesets, manifests, and files - each have unique requirements
  that will dictate special, one-off methods on their storage
  interfaces. I'd rather we define our interfaces so they are
  tailored to each type initially. If an implementation wants to
  shoehorn all data into generic key-value blog store, they can
  still do that. And we also reserve the right to combine interfaces
  in the future. I just think that attempting to have the initial
  versions of the interfaces deviate too far from current reality will
  make it very challenging to define and implement them.
  
  The reason I'm defining and implementing this interface now is to
  support new (experimental) wire protocol commands to be used to
  support partial clone. Some of these commands will benefit from
  aggressive caching. I want to prove out the efficacy of the interfaces
  approach by implementing cache-based speedups in the interface layer.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D2883

AFFECTED FILES
  mercurial/localrepo.py
  mercurial/repository.py
  mercurial/revlogstore.py

CHANGE DETAILS

diff --git a/mercurial/revlogstore.py b/mercurial/revlogstore.py
new file mode 100644
--- /dev/null
+++ b/mercurial/revlogstore.py
@@ -0,0 +1,37 @@
+# revlogstore.py - storage interface for repositories using revlog storage
+#
+# Copyright 2018 Gregory Szorc <gregory.szorc at gmail.com>
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+
+from __future__ import absolute_import
+
+from . import (
+    error,
+    filelog,
+    repository,
+)
+
+class revlogfilesstore(repository.basefilesstore):
+    """Files storage layer using revlogs for files storage."""
+
+    def __init__(self, svfs):
+        self._svfs = svfs
+
+    def resolvefilesdata(self, entries):
+        for path, node in entries:
+            fl = filelog.filelog(self._svfs, path)
+
+            try:
+                rev = fl.rev(node)
+            except error.LookupError:
+                yield 'missing', path, node, None
+                continue
+
+            if fl.iscensored(rev):
+                yield 'censored', path, node, None
+                continue
+
+            data = fl.read(node)
+            yield 'ok', path, node, data
diff --git a/mercurial/repository.py b/mercurial/repository.py
--- a/mercurial/repository.py
+++ b/mercurial/repository.py
@@ -266,3 +266,33 @@
 
 class legacypeer(peer, _baselegacywirecommands):
     """peer but with support for legacy wire protocol commands."""
+
+class basefilesstore(object):
+    """Storage interface for repository files data.
+
+    This interface defines mechanisms to access repository files data in a
+    storage agnostic manner. The goal of this interface is to abstract storage
+    implementations so implementation details of storage don't leak into
+    higher-level repository consumers.
+    """
+
+    __metaclass__ = abc.ABCMeta
+
+    def resolvefilesdata(self, entries):
+        """Resolve the fulltext data for an iterable of files.
+
+        Each entry is defined by a 2-tuple of (path, node).
+
+        The method is a generator that emits results as they become available.
+        Each emitted item is a 4-tuple of (result, path, node, data), where
+        the first element can be one of the following to represent the operation
+        result for this request:
+
+        ok
+           Successfully resolved fulltext data. Data field is a bytes-like
+           object.
+        missing
+           Data for this item not found. Data field is ``None``.
+        censored
+           Data for this revision is censored. Data field is ``None``.
+        """
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -52,6 +52,7 @@
     pycompat,
     repository,
     repoview,
+    revlogstore,
     revset,
     revsetlang,
     scmutil,
@@ -479,6 +480,8 @@
             else: # standard vfs
                 self.svfs.audit = self._getsvfsward(self.svfs.audit)
         self._applyopenerreqs()
+        self.filesstore = revlogstore.revlogfilesstore(self.svfs)
+
         if create:
             self._writerequirements()
 



To: indygreg, #hg-reviewers
Cc: mercurial-devel