[RFC] kbfiles: an extension to track binary files with less wasted bandwidth

[Augie Fackler]

On Fri, Aug 5, 2011 at 12:00 PM, Martin Geisler <mg at lazybytes.net> wrote:
> "Na'Tosha Bard" <natosha at unity3d.com> writes:
>
> Hi Na'Tosha,
>
> I'm back from vacation and slowly catching up on all the emails you guys
> have sent the last two weeks :-)
>
>> I am really thrilled about how well this project is moving along, as
>> well. Without a tool like this, Unity would not be using any DVCS (at
>> least not without having to have developed some similar system first).
>> As it is, this makes it possible for a company in the game development
>> industry (an industry where binaries on the order of several gigabites
>> is the norm -- at least we don't have any that big) to use Mercurial.
>> Without bfiles, our clone sizes would be increasing on the order of at
>> *least* several hundred megabytes or by some gigabytes every week.
>>
>> This tool really opens the doors for a whole new group of users.
>
> I have a bit of feedback from my client -- the one who made the snap
> extension. That extension is now being decommisioned and the department
> that used it is looking for an alternative solution.
>
> One problem they mentioned is the per-file overhead: Instead of having
> 200 files of 1 GB, they have 20,000 files, each of which is just 10 MB.
> That is kind of upside-down compared to what I expected. How many big
> files do you have in your repository?
>
> From skimming the code, it seems that kbfiles opens a connection per
> file that is sends to a remote store. If that is right, it sounds costly
> when the number of files grow very large.

It shouldn't be hard (for http, at least) to have some kind of
connection pool. I've written one already for our urlopen support
using mercurial.httpclient...

>
> --
> Martin Geisler
>
> Mercurial links: http://mercurial.ch/
>
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at selenic.com
> http://selenic.com/mailman/listinfo/mercurial-devel
>
>