Auto-refresh proposal

[Greg Ward]

On Wed, Feb 17, 2010 at 10:32 AM, Qi Yang <qi.yang137 at gmail.com> wrote:
> Sorry, there were some symbols messed up in last e-mail. I'd appreciate if
> anyone could give some feedbacks or suggestions on the following proposal.

Sorry for the delay.  I did read your message and have been thinking
about it.  My current thoughts follow...

> Given "auto-status" proposal is set
> (http://selenic.com/pipermail/mercurial-devel/2010-February/018892.html).
> I'd like to propose to directly call bfrefresh when doing commit in bfiles
> and make "bfrefresh" command private.

By "private", do you mean leave it as a command but don't expose it in
the extension help?  Or take it out of the command table entirely and
just leave it as an internal implementation detail?

> The benifit of doing so include the following:
>
> 1.       No worries about inconsistency between standin and bfile
>
>     * Standin and bfiles will be synchronized when commit. The only time
> when they are out of sync would be: a. when "hg up -r <revision#>" is called
> or b. when standin file is manually changed by user.

(a) will be alleviated by implementing auto-update, but we should not
count on that feature.  (Some people will no doubt turn it off, or it
might be broken.)  And we can't do anything about (b) except
acknowledge that it will happen, and we should assume that the user
knows what they are doing.  (Unless they demonstrate otherwise by
creating a syntactically invalid standin.)

Now, at first blush, I am inclined to say that in both cases the
standin file should take precedence over the big file in the working
dir.  But that risks losing the user's work.  Consider:

  * user modifies bigfile
  * user runs "hg update X" where X is a changeset with a different
revision of bigfile
  * user runs "hg bfupdate": if bfupdate gives the standin (modified
by "hg update") precedence, it will trash the user's modification to
bigfile: not good

Hmmm: we have the same problem if auto-update mode comes along and
integrates bfupdate into update.

Conclusion: auto-refresh mode does *not* relieve us of worrying about
inconsistency between standin and big file.

>     * In scenario a. we can either enable "auto-update" or give a warning
> message to user when "hg up" is called and "auto-update" is not enabled.

See above.  I don't think auto-update helps.  It might even make
things worse, since it would remove the window between "update" and
"bfupdate" where the user can discover the problem.

>     * In scenario b. we can detect if the standin file is modified by
> compare the mtime(standin) and the latest commit time.

I don't like relying on mtime.  Its resolution is too coarse and it
feels fragile.  Example: in my experience, dirstate is one of the
trickiest things in Mercurial, and it's largely because of the
vagaries of mtime.  (It took me a *lot* of time and hard work to get
bfiles' interactions with dirstate right.  And I believe Martin
Geisler has had similar difficulties in the eol extension.  It's
tricky stuff, and it's largely because of mtime's coarse resolution.)

IMHO the only reliable way to know if a file has been modified is to
compare its content.  That's why bfiles has so much state tucked away
inside .hg/bfiles: its own dirstate, latest/, pending/, and
committed/.  Each is there for a reason; nothing is arbitrary.

The good news is that there is already so much state in .hg/bfiles
that it might be possible to do a reliable "is it changed?" check
without adding any more state.  The bad news is that it requires
reading the content of the file to compute the hash.  OTOH, it's
entirely possible I've already implemented exactly the check you need.
 Take a hard look at _get_status(); it might already be what you want.

> 2.        Easier to detect when standin is modified by user
>
>     * There has been a lot of discussion about how to detect this problem
> (http://selenic.com/pipermail/mercurial-devel/2010-February/018539.html) if
> we this proposal is possible, we could simplify this problem and solve it as
> solution in scenario b.

I'm still not convinced this is a problem.  Yes, we need to do
something about corrupt standins.  (E.g. user forgets to resolve a
merge conflict and leaves a file with "<<<<" markers.)  But if I put a
valid SHA-1 hash in a standin that refers to a revision that already
exists somewhere (e.g. in .hg/bfiles/pending, .hg/bfiles/committed, or
the central store), then that should be accepted.

> From earlier discussion with Greg
> (http://selenic.com/pipermail/mercurial-devel/2010-February/018381.html),
> the major issue of eliminate bfrefresh is how the status is reported to
> users. Therefore, Integrating bfiles status, bfrefresh, as well as detecting
> if user adds/modifies/removes standin file should be combined. I propose the
> following solutions:
>
> 1.       Implement all these features in bfiles (it is mandatorily enabled)
>
> 2.       Allow user to enable/disable "auto-status", but "auto-bfrefresh" is
> mandatory (so that it is easier to detect standin changes). However, if
> disable "auto-status", the status report may be surprising.

I think mandatory auto-status is much less controversial than
mandatory auto-refresh.  Here's my thinking:

  * keep bfstatus as-is, i.e. it tells you everything about big file states
  * mandatory auto-status mode: i.e. always wrap status to show big
file states just like regular states using Anton's criterion: "hg
status" should show the same thing whether I "hg add" or "hg bfadd" a
file (N.B. this has to be done by wrapping either
localrepository.status() or dirstate.status() to work with TortoiseHg)
  * implement optional auto-refresh mode, ie. keep bfrefresh as a
visible command

If optional auto-refresh turns out to be fiendishly complex, and
making it mandatory would be a great simplification, then please try
again to convince me.  But I'm not convinced yet.  Idea: implement
optional auto-refresh mode and get that patch in.  Then prepare a
patch that makes auto-refresh mandatory and drops/hides bfrefresh, and
let's see how much complexity drops away.  If it's enough, then I'll
be convinced and I'll take the patch.  But if it's not very much, or
if it actually breaks things... no dice.

Greg