[RFC] kbfiles: an extension to track binary files with less wasted bandwidth

Martin Geisler mg at lazybytes.net
Fri Aug 5 12:00:13 CDT 2011


"Na'Tosha Bard" <natosha at unity3d.com> writes:

Hi Na'Tosha,

I'm back from vacation and slowly catching up on all the emails you guys
have sent the last two weeks :-)

> I am really thrilled about how well this project is moving along, as
> well. Without a tool like this, Unity would not be using any DVCS (at
> least not without having to have developed some similar system first).
> As it is, this makes it possible for a company in the game development
> industry (an industry where binaries on the order of several gigabites
> is the norm -- at least we don't have any that big) to use Mercurial.
> Without bfiles, our clone sizes would be increasing on the order of at
> *least* several hundred megabytes or by some gigabytes every week.
>
> This tool really opens the doors for a whole new group of users.

I have a bit of feedback from my client -- the one who made the snap
extension. That extension is now being decommisioned and the department
that used it is looking for an alternative solution.

One problem they mentioned is the per-file overhead: Instead of having
200 files of 1 GB, they have 20,000 files, each of which is just 10 MB.
That is kind of upside-down compared to what I expected. How many big
files do you have in your repository?

From skimming the code, it seems that kbfiles opens a connection per
file that is sends to a remote store. If that is right, it sounds costly
when the number of files grow very large.

-- 
Martin Geisler

Mercurial links: http://mercurial.ch/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://selenic.com/pipermail/mercurial-devel/attachments/20110805/f3e943d9/attachment.pgp>


More information about the Mercurial-devel mailing list