New bundle format; code structure

[Sune Foldager]

I'm currently looking at and thinking about the new bundle format. Not just the
format, but also how we'll implement it in the code. The current state of
affairs is a slight mess, as I wrote a mail about recently: sometimes full
bundles are used, often headerless ones. Sometimes there is compression,
sometimes not etc. I have tried to sum it up in the tables below:

Push:

proto    full bundle?    compression
http     yes             any (capability), unchunked
ssh      no              none, (streamed in 4k chunks)

Pull:

proto    full bundle?    compression
http     no              ext. zlib (4k chunks, 64k for decomp.)
ssh      no              none


Here "ext. zlib" means external zlib, as in: not being done by the bundler
(in general, not much bundle-related IS done by the two bundler classes..
they are a bit strange, really :p).

Since we're introducing a new argument for getbundle anyway, wouldn't it be
nicer if we could work towards something like this for pull:

Pull ng:

proto    full bundle?    compression
http     yes             any (capability/server's choice, chunked)
ssh      yes             any (capability/server's choice, chunked)

Default could then be zlib for http, none for ssh. Of course this will require
shuffling some code around, possibly enhancing the capabilities of the bundling
code and such, to ensure we can still do efficient chunked compressions and
decompressions. I have a vague idea in mind I could make more concrete.

For push, we need a new unbundle command (like getbundle; sendbundle?), and we
could perhaps "fix" ssh in that case as well, so we can unify it with http.

Thoughts on that?

Also, a draft of a new bundle format is here:
http://mercurial.selenic.com/wiki/BundleFormat2

It's rough, and Benoit had some comments (on irc), which I forgot. I tried to
include the following features:

- support for capabilities for future enhancements
- support for progress by including number of chunks for each changegroup
- support for deltaparent (for generaldelta)

Thoughts on that?

-Sune