Wire protocol futures

Gregory Szorc gregory.szorc at gmail.com
Tue Sep 4 12:19:41 EDT 2018


On Fri, Aug 31, 2018 at 10:44 PM Matt Harbison <mharbison72 at gmail.com>
wrote:

> I don't know much about the wire protocol, so this is mostly for my
> understanding...
>
> On Fri, 31 Aug 2018 18:47:34 -0400, Gregory Szorc
> <gregory.szorc at gmail.com> wrote:
>
> > Another problem that seemingly becomes simpler is large file storage. I
> > argue that largefiles and LFS today is effectively a hack to facilitate
> > non-partial clones despite the presence of large files. We store and
> > transfer flagged large files specially. But if your method of accessing
> > files data is through a dedicated "get file data" command, when you
> > squint
> > hard enough you realize that this is logically very similar to "all files
> > are using largefiles/LFS." This leads to questions like "if we have a
> > dedicated 'get file data' API, why do we need a special store / endpoint
> > for large files?" And if we communicate the sizes of files before file
> > data
> > is retrieved or don't transfer revision data over a size threshold unless
> > the client asks, this puts clients in the driver's seat about whether to
> > fetch large files revisions. We could implement all the benefits of
> > largefiles / LFS without it having to be a feature that repositories and
> > servers opt in to! i.e. clients could dynamically apply special storage
> > settings on large file revisions as they see fit.
>
> Interesting thought!  How does this downloading specific revisions on
> demand intersect with the append only nature of filelogs and (IIUC),
> entries being stored against the parent as deltas?  IOW, the client
> requests file at 100 (so store a full snapshot), and later file at 50, and
> then
> maybe file at 75.  Is this all rebuilt on the client side, or is this
> totally
> dependent on an alternate file storage?  Presumably this sort of thing is
> handled in RFL (I've never used it), but would it scale to truly large
> files?
>

Shallow clones and revlogs aren't the greatest match in the world. The way
it works in narrow is when we "backfill" a revlog, we strip it then replay
revisions in DAG order. This is because revlogs require children to occur
after their parents. This is sub-optimal in numerous ways.

We /could/ shoehorn large file storage into revlogs when shallow clones are
in play. In that case, we could always store fulltext revisions. Or we
could compute new deltas as appropriate. The recent work around more
flexible delta computation will be useful here.

But I think revlogs and shallow clones are so fundamentally incompatible
that we'll end up using an alternate storage backend for partial clones in
the long run. I assume alternate storage backends will still want to store
deltas for most revisions. For performance reasons, we likely default to
deltas emitted by the server (computing deltas is expensive). But I view
the "delta or fulltext" question and the exact semantics for that decision
(including for large revisions) to be the domain of the storage backend. If
e.g. the backend wants to store large blobs elsewhere (including on a
remote server), it could do that. The important thing is it fulfills its
storage backend APIs that allow it to obtain the fulltext data for a
revision when requested.


>
> > Let's talk about hashes.
> >
> > Mercurial uses SHA-1 for content indexing. We know we want to transition
> > off of SHA-1 eventually due to security weaknesses. One of the areas
> > affected by that is the wire protocol. Changegroups use a fixed-width 20
> > byte field to hold node values. That means we need to incur some kind
> of
> > BC
> > break in order to not use SHA-1 over the wire protocol. That's either
> > truncating a longer hashing algorithm output to 20 bytes or expanding the
> > fixed-width field to accommodate a different hash (likely 32 bytes).
> > Either
> > way, it requires a BC break because old clients would barf if they saw
> > data
> > with the new format.
> >
> > In addition, Mercurial has 2 ways to store manifests: flat and tree.
> > Unfortunately, any given repository can only use a single manifest type
> > at
> > a time. If you switch manifest formats, you change the manifest node
> > referenced in the changeset and that changes the changeset hash.
> >
> > The traditional way we've thought about this problem is incurring some
> > kind
> > of flag day. A server/repo operator makes the decision to one day
> > transition to a new format that hashes differently. Clients start pulling
> > the new data for all new revisions. Every time we talk about this, we get
> > uncomfortable because it is a painful transition to inflict.
> >
> > I think we can do better.
> >
> > One of the ideas I'm exploring in the new wire protocol is the idea of
> > "hash namespaces." Essentially, the server's capabilities will advertise
> > which hash flavors are supported. Example hash flavors could be
> > "hg-sha1-flat" for flat manifests using SHA-1 and "hg-blake2b-tree" for
> > tree manifests using blake2b. When a client makes a request, that request
> > will be associated with a "hash namespace" such that any nodes referenced
> > by that command are in the requested "hash namespace."
> >
> > This feature, if implemented, would allow a server/repository to index
> > and
> > serve data under multiple hashing methodologies simultaneously. For
> > example, pushes to the repository would be indexed under SHA-1 flat,
> > SHA-1
> > tree, blake2b flat, and blake2b tree. Assuming the server operator opts
> > into this feature, new clones would use whatever format is
> > supported/recommended at that time. Existing clones would continue to
> > receive SHA-1 flat manifests. New clones would receive blake2b tree
> > manifests. No forced transition flag day would be required. Server
> > operators could choose to keep around support for legacy formats for as
> > long as they deemed necessary. And the "changesetdata" command I'm
> > proposing could allow querying the hashes for other namespaces, allowing
> > clients to map between hashes.
>
> Would .hgtags and .hgsubstate be re-written on the fly, like convert
> does?  (Maybe .hgsubstate can't be, because those are really subrepo
> hashes.  Unless we take the approach that the subrepo clone inherits its
> parent's selection.)  If yes, then maybe the same mechanism can be used
> to
> rewrite any hashes in the commit message and extras?
>

These are all open questions. I haven't thought about them too much.
.hgtags could probably be rewritten automatically. .hgsubstate is a bit
harder. As are the other semantics around multi-hashing. Tons of corner
cases to work out.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20180904/8fe59616/attachment.html>


More information about the Mercurial-devel mailing list