[PATCH 3 of 4 tags-cache-split] tags: extract .hgtags filenodes cache to a standalone file

[Gregory Szorc]

On Tue, Mar 31, 2015 at 5:31 PM, Pierre-Yves David <
pierre-yves.david at ens-lyon.org> wrote:

>
>
> On 03/31/2015 06:19 AM, Matt Mackall wrote:
>
>> On Sun, 2015-03-29 at 23:14 -0700, Gregory Szorc wrote:
>>
>>> # HG changeset patch
>>> # User Gregory Szorc <gregory.szorc at gmail.com>
>>> # Date 1426392860 25200
>>> #      Sat Mar 14 21:14:20 2015 -0700
>>> # Node ID 9b396004b4338aa3e011cbf4f13800a3c55b1ab6
>>> # Parent  c19ef55be215f68a0955c49473c2e07b721998ff
>>> tags: extract .hgtags filenodes cache to a standalone file
>>>
>>> Resolution of .hgtags filenodes values has historically been
>>> a performance pain point for large repositories, where reading
>>> individual manifests can take over 100ms. Multiplied by hundreds
>>> or even thousands of heads and resolving .hgtags filenodes becomes
>>> a performance issue.
>>>
>>> Earlier work to split the tags cache into per-filter files
>>> helped alleviate many of the pain points. However, it didn't
>>> address the overall problem of redundant .hgtags filenode resolution.
>>> In fact, it introduced a new one: redundant .hgtags filenode
>>> resolution for each per-filter cache file.
>>>
>>
>> Again, please take a look at the new revbranchcache and either try to
>> parallel it or explain why you're doing something different. It's
>> already doing the same thing for rev->branch names that you're doing
>> for .hgtags nodes, compactly and quickly.
>>
>
> I've spend some time in VC with greg, to talk about how reb branch cache
> collaborate and how such approach could also be used by tags cache.
>
> Collaboration is now clearer in Greg minds while the way the tag cache is
> clearer in mind. There will be a new version of this series.
>

We talked about this again today at PyCon. Current plan:

1) Establish a shared file for the .hgtags -> filenode mapping that works
like rev branch cache (binary sparse-populated array)
2) Establish per-filter tags cache files
3) Change format of tags cache files to a) remove .hgtags fnodes b)
properly incorporate hiddens revs
4) (distant future) refactor the per-filter tags caches to interact better
to allow reuse, avoid extra computation, etc (similar to how branch head
caches work)

I'm going to try to do 1-3 during the sprint.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://selenic.com/pipermail/mercurial-devel/attachments/20150412/fff1a048/attachment.html>