[PATCH 1 of 6 V2] obsstore: add a 'cachekey' method

Pierre-Yves David pierre-yves.david at ens-lyon.org
Tue May 23 05:49:56 EDT 2017


Please ignore this email, I've unified it with the reply to patch 5, and 
then … I sent the wrong one.

On 05/23/2017 11:47 AM, Pierre-Yves David wrote:
>
>
> On 05/23/2017 04:22 AM, Jun Wu wrote:
>> I have got a chance to test this in production with one of our biggest
>> repo
>> and a 18MB obsstore. I tried running "hg id" with a a minimal hgrc (with
>> just lz4revlog and remotefilelog enabled) and without an existing
>> obscache.
>>
>> dualsourcecache.update takes 9 seconds. 7 seconds are used to scan
>> changelog, 0.36 seconds used to scan obsstore.
>>
>> The related profiling data is as below:
>>
>>   9735       | _computeobsoleteset              obsolete.py:1554
>>   9421        \ update                          obsolete.py:1227
>>   1872          \ _upgradeneeded                obsolete.py:1311
>>    434            \ revs (many times)           changelog.py:319
>>   1179            \ __iter__                    obsolete.py:564
>>   1179             | __get__                    util.py:798
>>   1177             | _all                       obsolete.py:707
>>   1138             | _readmarkers               obsolete.py:444
>>   1138             | _fm1readmarkers            obsolete.py:432
>>   7549          \ _updatefrom                   obsolete.py:1433
>> * 7185            \ _updaterevs                 obsolete.py:1439
>>    766              \ __get__                   util.py:798
>>    766               | successors               obsolete.py:717
>>    766               | _addsuccessors           obsolete.py:506
>> * 3964              \ node (many times)         changelog.py:359
>>    364            \ _updatemarkers              obsolete.py:1471
>>
>> So it is expensive for us to rebuild the cache. It seems the cache may be
>> dropped in some cases (ex. histedit --abort). I wonder if we can make the
>> cache more persistent so it'll be impossible to be nuked entirely,
>> like when
>> doing a strip, just truncate the cache file to the strip point instead of
>> nuking it.
>
> The simplest solution I can see here is to backup the cache when
> histedit/rebase starts and reinstall the cache when the operation is
> aborted. That way we can incrementally upgrade from that backup point.
> This approach can be generalized to other caches suffering from strip.
>
> Stripping the cache is also an option, but only in the case where only
> the changelog has been stripped (not the obsstore).
>
> (note: there are likely simple optimization lurking in the update
> function too. For example is len(obsstore) << len(repo), we could focus
> on iterating on the obsstore only when rebuilding the cache, etc…).
>
>
> Does these details address your concerns?
>
> Cheers,
>

-- 
Pierre-Yves David


More information about the Mercurial-devel mailing list