[PATCH 1 of 6 V2] obsstore: add a 'cachekey' method
Pierre-Yves David
pierre-yves.david at ens-lyon.org
Tue May 23 05:49:56 EDT 2017
Please ignore this email, I've unified it with the reply to patch 5, and
then … I sent the wrong one.
On 05/23/2017 11:47 AM, Pierre-Yves David wrote:
>
>
> On 05/23/2017 04:22 AM, Jun Wu wrote:
>> I have got a chance to test this in production with one of our biggest
>> repo
>> and a 18MB obsstore. I tried running "hg id" with a a minimal hgrc (with
>> just lz4revlog and remotefilelog enabled) and without an existing
>> obscache.
>>
>> dualsourcecache.update takes 9 seconds. 7 seconds are used to scan
>> changelog, 0.36 seconds used to scan obsstore.
>>
>> The related profiling data is as below:
>>
>> 9735 | _computeobsoleteset obsolete.py:1554
>> 9421 \ update obsolete.py:1227
>> 1872 \ _upgradeneeded obsolete.py:1311
>> 434 \ revs (many times) changelog.py:319
>> 1179 \ __iter__ obsolete.py:564
>> 1179 | __get__ util.py:798
>> 1177 | _all obsolete.py:707
>> 1138 | _readmarkers obsolete.py:444
>> 1138 | _fm1readmarkers obsolete.py:432
>> 7549 \ _updatefrom obsolete.py:1433
>> * 7185 \ _updaterevs obsolete.py:1439
>> 766 \ __get__ util.py:798
>> 766 | successors obsolete.py:717
>> 766 | _addsuccessors obsolete.py:506
>> * 3964 \ node (many times) changelog.py:359
>> 364 \ _updatemarkers obsolete.py:1471
>>
>> So it is expensive for us to rebuild the cache. It seems the cache may be
>> dropped in some cases (ex. histedit --abort). I wonder if we can make the
>> cache more persistent so it'll be impossible to be nuked entirely,
>> like when
>> doing a strip, just truncate the cache file to the strip point instead of
>> nuking it.
>
> The simplest solution I can see here is to backup the cache when
> histedit/rebase starts and reinstall the cache when the operation is
> aborted. That way we can incrementally upgrade from that backup point.
> This approach can be generalized to other caches suffering from strip.
>
> Stripping the cache is also an option, but only in the case where only
> the changelog has been stripped (not the obsstore).
>
> (note: there are likely simple optimization lurking in the update
> function too. For example is len(obsstore) << len(repo), we could focus
> on iterating on the obsstore only when rebuilding the cache, etc…).
>
>
> Does these details address your concerns?
>
> Cheers,
>
--
Pierre-Yves David
More information about the Mercurial-devel
mailing list