[RFC] kbfiles: an extension to track binary files with less wasted bandwidth

[Greg Ward]

On Sun, Aug 14, 2011 at 6:24 PM, Andrew Pritchard <awpritchard at gmail.com> wrote:
>> Idea: how hard would it be to add the standin prefix dir to the wire
>> protocol? It should only have to be sent once per conversation.
>
> That would actually not be particularly difficult, and it could easily
> be stored in '.hg/largefiles/standins' or the like.  The
> largefiles-kiln branch already has code for dealing with per-repo
> prefixes, to make kbfiles repositories continue to work, but it
> currently just checks for the existence of ''.hg/store/data/.hglf/*.i"
> or ".kbf/*.i".

So users of kbfiles have the same problem as users of bfiles? I.e.
switching to largefiles would be easy if largefiles transparently
recognized the old standin dir, but disruptive if we force them to
convert with a filemap?

> This would still make hosting providers' jobs a bit
> harder, since they would have to keep track of the prefix in order to
> handle largefiles separately.

If they choose to. IMHO if a hosting provider chooses to say "we
support largefiles, but *only* with the canonical default .hglf [or
whatever we settle on] prefix", that's entirely reasonable. bfiles and
kbfiles should be viewed as prototypes.

> Just brainstorming here, but currently the value of the 'lfilestore'
> capability carries no meaning: I set it arbitrarily to 'serve' to
> leave it open to future versions of the protocol or alternative ways
> of providing a store, but it could be used as the prefix, as in
> 'lfilestore=.hglf'.  This would be a bit weird in that the
> capabilities reported by the server could vary per repository, and it
> also doesn't solve the problem of identifying the prefix when pushing
> largefiles changesets to a repository that currently has none.

That does seem weird.

> Abandoning that line of thought almost entirely, it could use the
> pushkey mechanism, adding no new wireproto commands and even taking
> advantage of the warn-on-already-existing key behavior in the case of
> a repository that already has largefiles.

Yeah, that sounds sensible.

> There could be some weirdness if two developers simultaneously add
> largefiles to a previously-vanilla repository and try to push them,
> but as long as new largefiles repositories always use a specific
> prefix, that shouldn't cause any problems.

I don't think we should expose any way to use a non-default standin
prefix. .hgbfiles/ and .hgkbf/ are legacy prefixes that we should
support for the convenience of users, but there is no reason to create
a new largefiles repo using those prefixes. One ring to rule them all,
and all that.

Greg