[PATCH] localrepo: introduce persistent caching of revset revision's branch names

Gregory Szorc gregory.szorc at gmail.com
Wed Oct 15 13:02:22 CDT 2014


On 10/15/14 4:55 AM, Mads Kiilerich wrote:
> On 10/15/2014 06:53 AM, Gregory Szorc wrote:
>> Append only is a nice ideal and just that: an ideal. Things like
>> transaction rollbacks are effectively strips. And transaction
>> rollbacks can happen when e.g. a server-side hook rejects a push. And
>> if that hook (or a hook that ran before) accesses branch data and
>> causes a cache update that would trigger invalidation on rollback, the
>> next unsuspecting user triggers a fresh cache rebuild and experiences
>> extreme latency (if the repo is moderately sized).
>
> (It sounds like _that_ issue could be mitigated by disabling saving of
> caches while inside transactions and leave it to the next user to update
> the cache. A real solution would however be nice.)

How do you propose we do that? Innocuous read-only-looking code like 
repo.branches() and repo.tags() results in writing cache files. We've 
considered doing crazy things. We could even submit patches to Mercurial 
core. But I'm pretty sure marmoute would reject everything, telling us 
to implement the next generation of transactions instead. He would be right.

>> We see this at Mozilla with rejected pushes causing branchcache
>> rebuilds. On our Try repo, this leads to CPU exhaustion from multiple
>> clients all triggering cache generation simultaneously (because there
>> is no lock on the cache). This is why we spent time at the Summit
>> coming up with a better way to add non-revlog files into transactions.
>>
>> Whatever you do, please avoid full cache (re)populations wherever
>> possible.
>
> Sure, perfect would be good ... but also its enemy.

I'm not asking for perfect: I'm asking for a solution that will scale to 
large repositories. Mercurial scales for us today on all but 
repositories with thousands of heads. The eager cache invalidation 
demonstrated here patch smells like something that will reduce the 
scaling ceiling. (Benchmark the impact of a rollback to mozilla-central 
and mozilla-release to prove me wrong and I'll shut up.)

> If you have a plan for a general solution, shouldn't we use that instead
> of coming up with something special here? Until then, wouldn't it be
> better to give the lucky users the speed-up instead of giving everybody
> the worst case?

The general solution is fixing transactions. See the "Transactions" 
write-up on https://titanpad.com/mercurial32-sprint.



More information about the Mercurial-devel mailing list