[PATCH 04 of 15] streamclone: define first iteration of version 2 of stream format

[Boris Feld]

On Fri, 2018-01-19 at 15:54 -0500, Augie Fackler wrote:
> On Fri, Jan 19, 2018 at 09:08:48PM +0100, Boris Feld wrote:
> > # HG changeset patch
> > # User Boris Feld <boris.feld at octobus.net>
> > # Date 1516232936 -3600
> > #      Thu Jan 18 00:48:56 2018 +0100
> > # Node ID 4ee91fb55e208e8b139595ce9c2cae25aa9c54ea
> > # Parent  b80a8e39ac9bf984c25a666bd7f6c47d876d26af
> > # EXP-Topic b2-stream
> > # Available At https://bitbucket.org/octobus/mercurial-devel/
> > #              hg pull https://bitbucket.org/octobus/mercurial-deve
> > l/ -r 4ee91fb55e20
> > streamclone: define first iteration of version 2 of stream format
> 
> This is a good start of a series, but:
> 
> 1) patch 3 is begging for doctests on the varint scheme
> 

We are about to follow up with them.

> 2) This patch should probably include help/internals/ documentation
> on
>    the format, rather than only encoding it in a docstring
> 

We can follow up with that.

> 3) Patches that claim to be a big performance win, but don't include
>    any concrete testing numbers.

The performance win is about recomputing cache. We are getting number
as we speak, but performance issue related to branchmap and tags have
been well documented in the past.

On our repository with the most heads, it took 25 minutes to recompute
the tags cache and 1 minute for the branch cache. And the laptop wasn't
doing much else.

> 
> I think at this point we're going to admit defeat in the name of
> getting an RC release done before I go to bed tonight. Performance
> information I'd like to see in any v3 of this series:
> 
> 1) comparison between the new streaming clone and the existing one on
> small repos
> 
> 2) comparison on a medium repo with few branches (the hg repo could
> be good for this)
> 
> 3) comparison on a large repo with many heads (might need to use
> contrib/synthrepo to make something for this?)
> 
> 4) comparison on a large repo with many named branches (pypy?)
> 
> 5) comparison on mozilla-central

Those characteristics shouldn't impact the performance of the stream
clone.

> 
> I'm gathering that this is important work for you all for an
> important
> client or set of clients, and I'm sorry we're not going to manage it
> today. In the future, you could potentially sidestep some of this
> frustration by giving people a heads up earlier than a
> performance-critical 14 patch series with a new wireproto format.