statdaemon extension

[Martin Geisler]

Nicolas wrote some questions for me on Facebook:

> Interesting. Shouldnt be difficult to piggy back on inotify code to
> get linux & mac backends.

Exactly -- that's definitely the hope! I looked at the inotify and
FSEvents APIs and it seems that the "tell me the state of this
directory" API of the statdaemon fits both of them well.

One thing that could be better: for Windows, the daemon will listen on
events in the working copy -- recursively. This means that it will also
get events for (huge) ignore directories.

I did it like this since I was wary of making the daemon understand the
.hgignore file (and in particular understand updates to it) and since it
seems costly to begin monitoring each directory by itself.

As I understand it, you would use the WaitForMultipleObjects function to
wait on a file handle in Windows. That function can only wait for
MAXIMUM_WAIT_OBJECTS, which is normally only 64. Since the extension is
meant for file trees with many thousands of directories, one would need
to make some tree-like structure of objects to wait on. That seemed a
bit too cumbersome for a first prototype :)

> Something that really isnt clear is how you fix the never-ending sync
> issues. In particular:
>
> 1) when your StatServer replies to fetchall, you have no guarantee
> that the latest scan has completed. Which means that you're
> potentially replying with old data
>
> 2) when your listener modifies self._cache, your StatServer might be
> reading it. Which means that fetchall() replies might be reflecting
> some incomplete / inconsistent state
>
> 2) is easy to fix (Lock!)
>
> Mitigating 1) seems harder since you seem to be scanning all the time.
> Any way to reply "I'm sure there was no events since I received your
> message, and here's the state?"

I actually don't think it needs fixing. The way I see it, the daemon has
to reply with a consistent snapshot: that is, a snapshot that did exist
at some point in time. To be useful, the snapshot should preferably have
existed at some very recent point in time :)

So if lots of events are coming in and the daemon is busy updating its
cache, then it can still send out a copy of its cache: this cached
snapshot represents the state you would see if you could do 'hg status'
really, really fast and thus capture the half-updated working copy.

It might very well be that a subsequent 'hg status' will show more
changes -- but that's just because it was run at a later point in time.

What is not allowed to happen is that the daemon returns a cache
mentioning only

  ['bbb', 'xxx']

when whatever process is modifying the working copy did

  write('aaa')
  write('bbb')
  write('xxx')
  write('yyy')

The snapshot with only ['bbb', 'xxx'] is inconsistent, but any snapshot
with a "prefix" of ['aaa', 'bbb', 'xxx', 'yyy'] is okay.

This has some connection with the idea of "serializability" -- that the
final result will look *as if* the overlapping operations were executed
serially. Here a cache with ['aaa', 'bbb'] would be okay, since it would
be the result you would expect if the serial order had been:

  write('aaa')
  write('bbb')

  hg status

  write('xxx')
  write('yyy')

In real life the two processes were running in parallel, so it's
difficult to say what the true order was. But the ['aaa', 'bbb'] cache
is consistent.

If the event buffer overflows, the daemon will drop its cache. That
should also mean that the cache only ever contain correct and consistent
data.

-- 
Martin Geisler

aragost Trifork
Commercial Mercurial support
http://aragost.com/mercurial/