xrevlog: experimental reimplementation of revlog in C

[Adrian Buehlmann]

On 09.11.2010 15:25, Greg Ward wrote:
> Hi folks --
> 
> for ages now, it has really bugged me that Mercurial reads the entire
> changelog index into memory for almost any command, and that it
> creates a Python list *and* dictionary for it.

(Stupid questions below)

Why is this a problem?

It seems, even for my old netbeans repo clone with ~157,000 changesets I
have here for plying (.hg/store contains almost 2 GB), hg operations
seem pretty fast here (Windows 7, with warm file cache):

  $ hg -q version
  Mercurial Distributed SCM (version 1.7+14-158ca54a79cc)

  $ hg --time log -r 150000 -p > nul
  Time: real 1.677 secs (user 1.560+0.000 sys 0.125+0.000)

  $ hg --time log -r 150000
  changeset:   150000:efbef1a91108
  parent:      149991:b475a280704e
  parent:      149999:3f7531c31b29
  user:        ffjre at netbeans.org
  date:        Tue Oct 20 00:07:43 2009 +0200
  summary:     Merge of http://hg.netbeans.org/jet-main/

  Time: real 0.351 secs (user 0.109+0.000 sys 0.031+0.000)

  $ hg --time log -r efbef1a91108
  changeset:   150000:efbef1a91108
  parent:      149991:b475a280704e
  parent:      149999:3f7531c31b29
  user:        ffjre at netbeans.org
  date:        Tue Oct 20 00:07:43 2009 +0200
  summary:     Merge of http://hg.netbeans.org/jet-main/

  Time: real 0.853 secs (user 0.577+0.000 sys 0.062+0.000)

How much faster would this be with your xrevlog? Does it need to be faster?

> On our main repo at
> work, that's two 112,000-element containers every time you run (say)
> "hg tip".

That's 112,000 x 64 bytes = 6.7 MB index file which seems to be parsed
quite quickly by the C code in parsers.c (I guess that's function
parse_index there).

Why would mmap be a good fit for that? Simply parsing the index file in
one go seems good enough to me.

Even the 00changelog.i for my 157,000 entries netbeans clone is "only"
9.6 MB.