RFC: bitmap storage for precursors and phases

[Stanislau Hlebik]

Last year Durham sent a proposal for bitmap storage for computehidden() https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-September/087860.html.

It got a few useful comments, two most important comments:

  1.  Use bitmaps for lower-level data structures, for example, bitmap for public commits and bitmap for commits that are affected by obsolescence markers
  2.  Add cache validation checks

This is a new RFC that addresses these issues.

  1.  In the code, there is a cache only for precursors. Later I'm planning to add cache for public commits.
  2.  Cache validation key was added. It can be any key. For precursorcache I've chosen obsstore file size and mtime and cache validation key. For phasecache we can use hash of sorted phaseroots file since this file is usually small.


In a few places, I decided to delete cache entirely:

  1.  Caches are blown up if we receive obsmarkers via bundle2 or exchange._pullobsolete(). This is not necessary since cache won't be valid on the next run anyway because obsstore size and/or mtime was changed. We can change it.
  2.  Obsstore can store markers for node that are not present in the repo. Since bitmap is basically a dict from rev to boolean it can't store markers for revisions that are not in the repo. It doesn't cause issues unless missing node is received from bundle or during pull. In this case precursorcache doesn't recognize it as obsoleted. To fix it cache is deleted during changegroup.apply() if we receive a node that is in obsstore.successors dict.
  3.  Similar thing in bundlerepo - new obsoleted nodes may have been added, let's just not use precursorcache in bundlerepo.


The code is here: https://bitbucket.org/stashlebik/hg/commits/branch/hiddenbitmaps
It's not super clean yet, but all tests pass (except maybe for commit/code checks).
Please look and let me know what you think!


Thanks,
Stas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20170210/1554d1da/attachment.html>