D1856: wireproto: support for pullbundles

[indygreg (Gregory Szorc)]

indygreg added a comment.


  In https://phab.mercurial-scm.org/D1856#31522, @durin42 wrote:
  
  > I'm too rusty on bundle2 at the moment to grok what magic would be required to pre-compress payloads.
  
  
  The ideal solution would be a way to //reset// the context for the byte stream. Essentially we'd add a marker telling consumers they've reached EOF of either a bundle2 stream or a compression context. The next byte should be interpreted as a new bundle2 stream or a new compression context.
  
  In the case of zstandard, we can force the end of a zstandard frame and start sending a new frame. As long as the consumer reads all frames as one coherent stream, the output of the decompressor will look as if nothing special happened. I'm not actually sure if python-zstandard supports this though. But it could. For other compression formats, the answer isn't as clear cut.
  
  It's probably best to have an explicit marker in the bundle2 or protocol stream identifying the end of a compression context. Either a "reset at end of part" flag in the part header. Or an explicit bundle2 part that says "reset after me." Either way, we'd need to invent an API on the compression engine that allows the compression context to be //reset//. This code could get gnarly, as there are various buffers in play. It's possible. But a bit of work. Since we're at code freeze and it looks like this feature will have to wait for 4.6, it looks like we'll have time to implement this feature if we want to go that route. I don't like scope bloating though.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1856

To: joerg.sonnenberger, #hg-reviewers, indygreg
Cc: indygreg, durin42, mercurial-devel