Making chg stateful

Jun Wu quark at fb.com
Thu Feb 2 09:34:47 UTC 2017


This is mainly to discuss more details about chg repo preloading idea [1].

Perf Numbers

  I wrote a hacky prototype [2] that shows significant improvements on
  various commands in our repo:
  
                               Before   After (in seconds)
      chg bookmark             0.40     0.08
      chg log -r . -T '{node}' 0.50     0.08
      chg sl                   0.71     0.27
      chg id                   0.71     0.37
  
  And hg-committed (with 12M obsstore) could benefit from it too mainly
  because the obsstore becomes preloaded:
  
                               Before   After
      chg bookmark             0.56     0.06
      chg log -r . -T '{node}' 0.56     0.07
      chg log -r . -p          0.86     0.08
      chg sl                   0.84     0.21
      chg id                   0.54     0.06
  
  So I think it's nice to get a formal implementation upstreamed. It's also
  easier to solve the perf issues individually like building the hidden
  bitmap, building an serialization format for the radix tree, etc.
  Although we can also build them later.

Stateful?

  Stateful is to the chg master server process.

  Currently the chg master process is stateless, after forking, the client
  talks to the forked worker directly, without affecting the master, and the
  worker does not talk back to master too:

    client         master          worker
      | connect() -> | accept()
      :              | fork() -----> |
      | send() --------------------> | recv()
      # client and worker no longer talk to master

  Therefore the master is currently stateless - it can only preload
  extensions but nothing about the repo state.

  The general direction is to make the worker tells the master what needs to
  be preloaded (like repo paths), and the master has a background thread
  preloading them. Then at fork(), the new worker will get the cache for
  free.

How about just preloading the repo object?

  There was a failed experiment: [3].

  The repo object depends on too many side-effects, and gets invalidated too
  easily. chg will not run uisetup()s, so it'll become harder to get a
  proper repo object.

So what state do we store?

  {repopath: {name: (hash, content)}}. For example:

    cache = {'/home/foo/repo1': {'index': ('hash', changelogindex),
                                 'bookmarks': ('hash', bookmarks),
                                 .... },
             '/home/foo/repo2': { .... }, .... }

  The main ideas here are:
    1) Store the lowest level objects, like the C changelog index.
       Because higher level objects could be changed by extensions in
       unpredictable ways. (this is not true in my hacky prototype though)
    2) Hash everything. For changelog, it's like the file stat of
       changelog.i. There must be a strong guarantee that the hash matches
       the content, which could be challenging, but not impossible. I'll
       cover more details below.

  The cache is scoped by repo to make the API simpler/easy to use. It may
  be interesting to have some global state (like passing back the extension
  path to import them at runtime).

What's the API?

  (This is an existing implementation detail. I'm open to any ideas)

  For example, let's say we want to preload the changelog index (code
  simplified so it does not take care of all corner cases).

  First, tell chg how to hash and load something:

      from mercurial.chgserver import repopreload

      @repopreload('index')
      def foopreloader(repo):
          # use the size of changelog.i as the hash. note: the hash function
          # must be very fast.
          hash = repo.svfs.stat('00changelog.i').st_size
          # tell chg about the current hash. if hash matches, the generator
          # function stops here.
          yield hash

          # if hash mismatches, load the changelog in a slower way.
          with repo.svfs('00changelog.i') as f:
              data = f.read()
              hash = len(f)
              index = _buildindex(data)
              index.partialmatch('ffff') # force build the radix tree
              # tell chg about the loading result and the hash that
              # absolutely matches the result.
              yield index, hash

  Then, repo.chgcache['index'] becomes available in worker processes. When
  initializing the changelog index, try to use the chg cache:

      # inside changelog's revlog.__init__:
      # note: repo.chgcache is empty for non-chg cases, a fallback is needed
      self.index = repo.chgcache.get('index') or _loadindex(...)

  The API is the simplest that I can think of, while being also reasonably
  flexible (for example, we can add additional steps like forcing building
  the radix tree etc). But I'm open to suggestions.

Some implementation details

  (This is the part that I want feedback the most)

  1) IPC between chg master and forked worker

     This is how the worker tells the master about what (ex. repo paths) to
     preload.

     I think it does not need to be 100% reliable. So I use shared memory in
     the hacky prototype [2]. Pipes are reliable and may notify master
     quicker, while have risks of blocking. shm seems much easier to
     implement and I think the latency of master getting the information is
     not a big deal. I prefer shm, but other ideas are welcomed.

  2) Side-effect-free repo

     This is the most "painful" part for the preloading framework to be
     confident.

     The preload function has a signature that takes a repo object. But chg
     could not provide a real repo object. So it's best-effort.

     On the other hand, most preload functions only need to hash file stats,
     i.e. just use repo.vfs, repo.svfs etc. They do not need a full-featured
     repo object.

     Therefore I think the choices are:

      a) Provide a repo object that is: localrepository - side effects
         (maximum compatibility)
      b) Build a fresh new repo object that has only the minimal part, ex.
         vfs, svfs etc. (less compatible)
      c) Do not provide a repo object. Just provide "repo path".
         (move the burden to the preload function writers)

     I currently prefer a). The plan is to move part of "localrepository" to 
     a side-effect free "baserepository" and use "baserepository" in the
     background preloading thread.

  3) Side effect of extensions

     The new chg framework will not run uisetup()s. Where the preloading
     framework does sometimes depend on some side effects of extensions'
     uisetup()s. For example, the *manifest extension could change greatly
     about what the manifest structure so the default manifest preloading
     won't work as expected.

     I think there are different ways to address this:

      a) Add a top-level "chgsetup()" which only gets called by chg, and is
         meant to be side-effect free.
         
         Extensions could wrap chg's pseudorepo object, and also register
         its own preloading functions.

         This is actually pretty clean. But I'm not sure whether "chgsetup"
         is a good name or not, or if a top-level function is a good idea in
         general.

      b) Have a config option to force chg to load extensions as it does
         today.

         The problems are uisetup will accept wrong ui objects, which just
         confuses developers (and is the motivation of the ongoing
         refactoring). And it probably makes chg's logic more complex than
         ideal.

     I cannot think of other good ideas on this. In this situation, I prefer
     a).

The rough plan

  I'd like to get the preloading API done. And then add logic to preload
  various things in another hgext. Preloading changelog index could be done
  with confidence. While other things could be a bit risky as extensions are
  unpredictable. Therefore a config option per repo to control what to
  preload is necessary.

[1]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-December/091846.html
[2]: https://bpaste.net/show/0dd5889cb453
[3]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-March/081615.html


More information about the Mercurial-devel mailing list