[PATCH 2 of 2 STABLE] store: add auto detection for hardlink count blindness (issue1866)

[Adrian Buehlmann]

On 01.11.2010 14:55, Adrian Buehlmann wrote:
> On 01.11.2010 14:17, Frank A. Kingswood wrote:
>> Matt Mackall wrote:
>>> On Sun, 2010-10-31 at 08:32 +0100, Adrian Buehlmann wrote:
>>>> store: add auto detection for hardlink count blindness (issue1866)
>>>
>>> - Test function is too heavy-weight - do the simplest fastest test
>>> possible. Probably something like:
>>>
>>>  f, fn = getasecurefilehandleandname(path)
>>>  fn2 = fn + "l"
>>>  link(fn, fn2)
>>>  stat
>>>
>>> Even this is pretty expensive.
>>
>> The patch calls it a lazy test but why not make it even lazier by 
>> putting it into util.copyfile? That way the cost is near zero - just one 
>> or two stat calls to check the link counts the first time a link is created.
>>
>> Would that not work?
> 
> I don't think so.
> 
> The problem happens for example when committing or pushing to a repo that
> is stored on a Windows share served by, let's say a Windows computer A, if
> that share is mounted on a Linux computer B or the share accessed from
> another Windows computer C.
> 
> If that same repo is accessed locally on A, then that repo looks differently.
> The hardlinks can be detected. But not if looking at that same repo from B or C.
> 
> If you would do a test when creating that repo, the result of the test would
> depend on where you executed it. It would not be the same on A as on B or C.
> Caching the result of the test done on A and then using it on B or C would be
> wrong.
> 
> My patch calls it a lazy test because the test is done at most once per opener
> instance on the first write. So, on a typical commit, the test is done once in
> .hg/store.
> 
> Here is an example of a commit with debug prints in the code:
> 
> $ hg ci -m1
> opener(Y:\a\.hg): atomictemp wb Y:\a\.hg\branchheads.cache
> opener(Y:\a\.hg): wb Y:\a\.hg\last-message.txt
> opener(Y:\a\.hg): wb Y:\a\.hg\journal.dirstate
> opener(Y:\a\.hg): wb Y:\a\.hg\journal.branch
> opener(Y:\a\.hg): wb Y:\a\.hg\journal.desc
> opener(Y:\a\.hg\store): detected hardlink blindness! -> setting _forcecopy
> opener(Y:\a\.hg\store): forced COW for Y:\a\.hg\store\data/a.txt.i
> opener(Y:\a\.hg\store): a+b Y:\a\.hg\store\data/a.txt.i
> opener(Y:\a\.hg\store): forced COW for Y:\a\.hg\store\00manifest.i
> opener(Y:\a\.hg\store): a+b Y:\a\.hg\store\00manifest.i
> opener(Y:\a\.hg\store): forced COW for Y:\a\.hg\store\00changelog.i
> opener(Y:\a\.hg\store): a+b Y:\a\.hg\store\00changelog.i
> opener(Y:\a\.hg\store): forced COW for Y:\a\.hg\store\00changelog.i
> opener(Y:\a\.hg\store): ab Y:\a\.hg\store\00changelog.i
> opener(Y:\a\.hg): atomictemp wb Y:\a\.hg\dirstate
> 
> Since only the store's opener has linktest set, only that opener will do the
> hardlink blindness test (and only in '.hg/store', while it is holding a write
> lock on the store).
> 
> In the above example, the test is triggered by writing to
> Y:\a\.hg\store\data/a.txt.i. It detected hardlink blindness, which causes a
> forced COW (=copy on write) on every write to a file in the store.
> 
> Writing to Y:\a\.hg\last-message.txt or creating the atomic temp for
> Y:\a\.hg\branchheads.cache will not trigger the test, because it is not a
> store opener which is doing the write.

BTW, the really costly thing is doing a copy of the files on every
write, if hardlink blindness has been detected (of course that happens
only if someone is pushing or committing to a repo stored on a Windows
share. Which doing so is lame by itself, so the user should not be
surprised if he gets lame speed as well. But it must not corrupt his repo).

In theory, we could additionally trade memory for speed if the opener
would cache the names of the files it has already copied (relative path
of the file to the base dir of the opener). This would save us from
copying '00changelog.i' a second time in the above example (optimizing
the lame case).