[RFC] kbfiles: an extension to track binary files with less wasted bandwidth

[Greg Ward]

On Sat, Oct 1, 2011 at 7:57 PM, Benjamin Pollack <benjamin at bitquabit.com> wrote:
>
> On Sep 28, 2011, at 6:04 PM, Greg Ward wrote:
>
>> I have two requests:
>>
>>  * [essential] make it possible for largefiles to seamlessly use repos
>>    where the standins live in .hgbfiles, so that existing users of
>>    bfiles don't have to go through a painful conversion process
>
> I will continue to argue that idea is misguided.
>
> The main point, as we've discussed at length previously, is that it results in bad behavior [1].  To recap: when you push, largefiles looks for largefiles in the changesets you're pushing, asks the server whether it has them already, and uploads any that the server doesn't already have.  It doesn't do any checking for changesets it's not pushing, since, if you're only ever using largefiles, changesets already on the server *must* have their corresponding largefiles.  For the common case I'm envisioning clients doing here--simply setting the largefiles prefix to .bfiles--this precondition is invalid, and you'll trivially be able to make repositories that no one can actually clone.

But I don't want to *configure* the standin dir. That was my original
proposal, but then Matt said "why not autodetect?" and I immediately
slapped my forehead and said "duh, yes, of course". Can't remember if
I said so publicly.

Here's the behaviour I want:

  if '.hglf/' is a dir:
    use that as the largefiles standin prefix
  elif '.hgbfiles' is a dir:
    use that as the largefiles standin prefix
  elif '.kbf' is a dir:
    use that as the largefiles standin prefix
  else:
    no largefiles in this changeset: fallback to 'hglf' in case
    someone adds one

Obviously that has to be done pretty early in setting up the
extension, but it can't be a constant anymore. I haven't looked at the
code enough to know if this is a straightforward refactoring or an
"OMG, don't go there" thing.

> Especially given that point, I remain unclear why we'd allow changing the largefiles prefix when we don't allow changing the name of .hg/.hgignore/.hgeol and so on.  If, for your particular setup, you *know* that the largefiles preconditions are actually satisfied, and you really don't want to reconvert, it's trivial to add a piggyback extension that swizzles out the standin prefix.  That's what we may do for grandfathering in .kbf standins for existing Kiln clients, for example.  But I really don't believe that behavior should be in core Mercurial.

There's no way in hell I'm going to convince my manager and 40
developers that we need to convert our entire repository (120k
changesets, with stuff in .hgbfiles/ dating back to 2002 -- this was
converted from CVS) just so I can stop maintaining bfiles and switch
us to largefiles. Who's going to update all the changeset ID
references in our Bugzilla comments? Or our build database? *shudder*
The only way that's gonna happen is if I can make it a seamless
transition.

Repeat that gripe for all other users of bfiles out there. I don't
know how many there are, but there's more than just one of us!

However, a silly little piggyback extension might be a tolerable
compromise. I'll keep it in my back pocket. Now... what to call it...
I know! How about "bfiles"? I'd love to see that monster shrink down
to a 5-line monkeypatch.

Greg