[PATCH 0 of 3] Transactional support for rebase/strip to avoid permanent repository corruptions

[Henrik Stuart]

Benoit Boissinot wrote:
> On Wed, Apr 15, 2009 at 10:34:29PM +0200, Henrik Stuart wrote:
> 
> [Thanks for the detailed explaination, I'll have a closer look at the patches later]
> 
>> Furthermore, since localrepository.transaction defines an "after" function
>> that renames journal* to undo*, and if the second call to transaction
>> returns the first transaction, then the "after" function will be called
>> again, but now there is suddenly no journal file(s) to rename anymore.
>> This is solved by, in addition to querying whether there is a weakref, to
>> see whether the transaction is running - if it is not, a new transaction
>> is returned.
>>
>> The alternative would have been to delete the transaction inside repair.strip,
>> but this would mean that we would never allow strip to be called nested
>> inside another transaction, something that we were not comfortable with
>> just decreeing (addchangegroup has this limit currently, but is it
>> intentional?).
> 
> I looked at the nesting transaction stuff the other day, and I think the
> only use of the nesting of in mq.
> Should we try to avoid it?

There should be no need to avoid nested transactions. Transactions
should, however, be used very, very carefully. Consider e.g. the
following case:

We have a repository with three revisions with a new file in each
revision and no other changes:

0    1    2
@----o----o
a    b    c

now we run "hg strip 1" and for some reason b.i fails at opening or
truncating.

If we had naïvely used the following pattern in repair.strip:

tr = repo.transaction()
offset = len(tr.entries)
cl.strip(striprev, transaction=tr)
repo.manifest.strip(striprev, transaction=tr)
for name in files:
  f = repo.file(name)
  f.strip(striprev, transaction=tr)
# do actual truncate here

then the changelog, manifest, and a's revlog files would have been added
to the journal (and thus also have been written and flushed to
.hg/store/journal as it should be), but repo.file('b') will raise an
IOError causing the strip to fail, however, there is now a partially
correct journal on disk and next time we run a repository-changing
command (that uses a transaction) we will be told to run "hg recover",
and when we do that: changelog, manifest, and a will be stripped, but b
and c will not, giving us a corrupted repository.

The correct thing to do is to make sure that your entire workload is
written such that there are no exceptions happening between adding the
first part of your payload and the final part of your payload. This is
why in the submitted patch that the above code is restructured to fetch
all the file revlogs before any truncation points are written to the
journal.

Strictly speaking, it could, hypothetically, be possible that the
journal write fails (disk full, too many corrupted sectors, etc.) also
causing a partial journal to be on disk. There is, again, no easy way to
fix this without changing the journal format. The way it would normally
be done would be to have a file format like this:

begin transaction
file1\0pos1
file2\0pos2
...
commit transaction

If there is no commit written, the journal block is incomplete and that
transaction cannot be replayed. This should be fairly analogous to how
most database servers handle this problem.

It /should/ be fairly esoteric cases that a partial journal file can be
written (modulo programmer errors in doing operations that may raise
errors inside their payload writing), but it can still happen, and I'd
like to see the journal format changed to the format above (or something
similar). Unless someone else gets to it first, I will probably submit
some further patches on this later.

That was a very long, tangential answer to a short question. :o)

-- 
Kind regards,
  Henrik Stuart