[PATCH 3 of 5] extdata: add basic caching

[Gregory Szorc]

On Fri, Sep 23, 2016 at 3:23 PM, Pierre-Yves David <
pierre-yves.david at ens-lyon.org> wrote:

>
>
> On 09/22/2016 08:21 PM, Matt Mackall wrote:
>
>> # HG changeset patch
>> # User Matt Mackall <mpm at selenic.com>
>> # Date 1474294391 18000
>> #      Mon Sep 19 09:13:11 2016 -0500
>> # Node ID 133b35066bef5d9c31d13c4f0b2e4a50d1ceae87
>> # Parent  9c8847df32a0c5045e60aded2e03a9c97507f909
>> extdata: add basic caching
>>
>> Sources used for log templating will be queried for every changeset
>> printed. So we need a cache. We attach the cache to the repo object to
>> give it a bounded lifetime.
>>
>> diff -r 9c8847df32a0 -r 133b35066bef mercurial/scmutil.py
>> --- a/mercurial/scmutil.py      Mon Sep 19 09:05:00 2016 -0500
>> +++ b/mercurial/scmutil.py      Mon Sep 19 09:13:11 2016 -0500
>> @@ -1453,6 +1453,14 @@
>>          raise error.Abort(_("extdata doesn't support parameters yet"),
>>                            hint=_("use double % for escaping"))
>>
>> +    # we cache external data sources for the lifetime of a repo object
>> +    # users like log templates may consult a data source very frequently
>> +    if not util.safehasattr(repo, "_extdatacache"):
>> +        repo._extdatacache = {}
>> +    cache = repo._extdatacache
>> +    if spec in cache:
>> +        return cache[spec]
>> +
>>
>
> While caching is obviously necessary, repo-life caching is likely to
> provide bogus result with long lived process like chg or the command
> server. We probably needs more aggressive invalidation. What about caching
> the data for each "commands" as a start.
>
> (that cache question is vast, because in some case we probably want to
> have a very strong caching with on disk version, but that is adventure for
> another castle imho)
>
>
We definitely need a more robust caching mechanism because this definitely
breaks hgweb, which uses hg.cachedlocalrepo to represent cached repos
between HTTP requests and currently only knows to invalidate if {changelog,
phaseroots, obsstore, bookmarks} files change.

In the current state of this patch, a repo._extdatacache entry will persist
for the lifetime of a hgweb process until the repo is written to. That
could be hours. At the very least you'll probably want to teach
cachedlocalrepo.fetch() to clear repo._extdatacache.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160923/4bfb8e89/attachment.html>