Mercurial and very large files...

[Tim Post]

On Tue, 2007-03-27 at 11:38 +0200, Marcin Kasperski wrote:
> I tried loading 1GB file to the mercurial repo. Killed the 
> process when my machine started to swap extensively.
> Then noticed the following sentence in FAQ:
> 
>    Mercurial currently assumes that single files, indices, 
>    and manifests can fit in memory for efficiency.
> 
> How deep is this restriction? Is it inherent for mercurial
> implementation, or something likely to be changed in
> the future?
> 
> Also: are there any suggestions, what is the max file
> size mercurial can be expected to work efficiently with?
> Of course this depends on the RAM size and CPU, but 
> is the dependency linear, or worse?
> 
> [in case somebody is curious: I am trying to manage
>  PGN files (chess games databases). Some of them
>  are large.]
> 
> PS Interesting that Git seems to be almost equivalent 
> to mercurial here. Experimenting, I just tried commiting 450MB 
> file. hg allocated ~900MB while commiting (I killed this process
> after some time). git-update-index allocated ~900MB while 
> execting git add. 
> 
> Git does not seem significantly better. git-update-index 
> just allocated 900MB while adding 600MB file...

Don't just think about your own box when doing this, assume a userland
of 512 MB slow DDR with 2 GB swap to slow SATA/IDE. I don't think you
could do large files and still be 'kind' to everyone participating.

Its better to just use wget or similar for that, with any system
(mercurial or otherwise). I'm new to mercurial but some limits are
universal. This falls under the same category as gravity.

Best,
--Tim