Transplant and dirstate

[Greg Ward]

I'm trying to debug a problem with transplant.  This is following on
from my question yesterday about concatenating history: transplant
works beautifully, up to the point where it crashes.  Hence the
debugging.  ;-)

Rough scenario: I've got repos foo1, foo2, and foo3 which each track
the history of the same set of files (different versions of the same
upstream package).  I'm trying to concatenate them into repo foo:
roughly, clone foo1 to foo, then transplant most of foo2 in, then
transplant most of foo3 in.  There is some additional fun and games
because each of foo1, foo2, and foo3 starts with an "import external
sources" commit and ends with "remove everything since we're no longer
using this version of foo".

The crash happens because transplant fails to transplant one of the
changesets in foo3 -- repo.commit() does nothing and returns None.
transplant fails to notice this and adds a (None, <src cset id>) entry
to its list of transplant pairs.  Then, trying to write
.hg/transplants/transplants, it calls node.hex(None) -- kaboom!  The
actual crash is in transplants.write() line 51, if you're following
along.  But that's a red herring.  The real problem is much earlier.

For example, why does transplant ignore repo.commit() returning None?
At the very least it should warn, possibly abort.  My current patch
includes this:

--- a/hgext/transplant.py
+++ b/hgext/transplant.py
[...]
@@ -246,9 +255,12 @@
             m = match.exact(repo.root, '', files)

         n = repo.commit(message, user, date, extra=extra, match=m)
+        if not n:
+            self.ui.warn('WTF?! nothing committed!! (n=%r)\n' % n)
         if not merge:
             self.transplants.set(n, node)

I now see that warning every time transplant crashes.  (This is a
"crash coming soon" warning, so it might as well raise Abort rather
than warn.  Also, when this happens, transplant silently drops a
changeset on the floor with no notice to the user apart from the crash
at the end.  Again, IMHO that's a good argument for aborting.)

But even that's not the real problem: why does commit() do nothing?
Because no files appear to have changed.  transplant trusts
patch.patch() to do the right thing and return a useful dict of
patched files. Heck, so do I.  Mercurial is pretty good at applying
patches, so I assume that patch.patch() works well 99.99% of the time.

So again, why does commit() do nothing?  Easy: because dirstate (via
repo.status()) tells it has nothing to do.  I demonstrated by calling
repo.status() right after patch.patch(), and it reports clean when
we're about to crash.  Clean status -> no commit -> append (None,
source) to transplant list -> crash calling hgnode.hex() while writing
.hg/transplants/transplants.

Here's where I get a little confused.  I have noticed two things in
tracking this problem down:

  * sometimes, when transplant crashes, it leaves my working dir in a
weird state: the working dir parent is the tip from *before* my
transplant run, and 'hg status' reports a ton of changes: apparently
the cumulative changes from all the transplanted changesets

  * the transplant crash is not 100% reliable; maybe 1 time in 10 it
doesn't happen and the transplant works fine

Both of these smell like dirstate bugs in transplant.  So I tried a
stab in the dark patch: save the dirstate after each transplant, i.e.
at the end of transplanter.applyone():

--- a/hgext/transplant.py
+++ b/hgext/transplant.py
@@ -249,6 +258,7 @@
         if not merge:
             self.transplants.set(n, node)

+        repo.dirstate.write()
         return n

     def resume(self, repo, source, opts=None):

And what-do-you-know, it worked.  transplant no longer drops source
changesets on the floor and no longer crashes writing
.hg/transplants/transplants.  And I suspect it will fix the "crashed
transplant leaves working copy in strange state" bug that I also saw.
Not bad for a one-liner.

But I don't entirely understand the fix!  If we're operating entirely
in memory, transplanting a long series of changesets from foo3 into
foo, it should not matter that dirstate on disk is out-of-date.
Shouldn't the in-memory dirstate be enough?

If anyone can explain why this patch works, I'll happily send it in.
But what if it's masking an even deeper bug?

Oh yeah, the bug is the same in latest crew-stable and latest crew.
Have not tried older versions.

Greg