<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Jul 4, 2016 at 8:48 AM, Simon Farnsworth <span dir="ltr"><<a href="mailto:simonfar@fb.com" target="_blank">simonfar@fb.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""># HG changeset patch<br>
# User Simon Farnsworth <<a href="mailto:simonfar@fb.com">simonfar@fb.com</a>><br>
</span># Date 1467646999 25200<br>
# Mon Jul 04 08:43:19 2016 -0700<br>
# Node ID 31eea7188911ca17b36e69170dfcec07f4da37ae<br>
<span class=""># Parent fd93b15b5c30d16fd9c9eba61402d07fc4085db3<br>
debug: automate the process of truncating a damaged obsstore (issue5265)<br>
<br>
We occasionally see users who've had a system crash damage the obsstore file<br>
in their .hg/store directory; this makes all `hg` commands fail until we go<br>
in and remove the damaged section of the obsstore by hand.<br>
<br>
Automate the process we use when this happens, as a debug command because it<br>
loses the corrupted data. We only use it in rare circumstances when it's<br>
important to retrieve a user's work and apply it to a fresh clone.<br>
<br>
diff --git a/mercurial/commands.py b/mercurial/commands.py<br>
--- a/mercurial/commands.py<br>
+++ b/mercurial/commands.py<br>
</span>@@ -15,6 +15,7 @@<br>
import re<br>
import shlex<br>
import socket<br>
+import struct<br>
import sys<br>
import tempfile<br>
import time<br>
@@ -3702,6 +3703,61 @@<br>
<div><div class="h5"> displayer.show(repo[r], **props)<br>
displayer.close()<br>
<br>
+@command('debugtruncatestore',<br>
+ [('', 'obsolete', None, _('truncate bad markers in obsstore'))],<br>
+ _('[OPTION]'))<br>
+def debugtruncatestore(ui, repo, **opts):<br>
+ """Fix up repository corruption by truncating damaged files<br>
+<br>
+ Most on-disk data structures are designed to be append-only. A failed write<br>
+ (e.g. due to an unexpected power failure) can leave the file corrupted.<br>
+<br>
+ This command attempts to recover from that situation by replacing the<br>
+ corrupted file with a version that only contains the valid records from the<br>
+ broken file. It is not guaranteed to remove all corrupt records - it will<br>
+ only remove corrupt records where normal use of the repo would result in a<br>
+ crash.<br>
+<br>
+ Corrupt files will be renamed with a .corrupt extension before the fixed<br>
+ version is written out, so that you can examine the corruption and/or undo<br>
+ this command.<br>
+<br>
+ You should normally use :hg:`recover` before resorting to this command.<br>
+ """<br>
+<br>
+ if 'obsolete' in opts:<br>
+ data = repo.svfs.tryread('obsstore')<br>
+ if data:<br>
+ # Slow algorithm - but this is an emergency debug operation<br>
+ version = None<br>
+ corrupt = False<br>
+ while version is None and len(data) > 0:<br>
+ try:<br>
+ (version, markers) = obsolete._readmarkers(data)<br>
+ # Force evaluation of all the markers - the pure<br>
+ # implementation returns a generator which won't detonate<br>
+ # until you evaluate the bad marker.<br>
+ for marker in markers:<br>
+ pass<br>
</div></div>+ except (ValueError, struct.error, error.Abort):<br>
<span class="">+ corrupt = True<br>
+ version = None<br>
+ data = data[:-1]<br>
+ continue<br>
+ break<br>
+ if corrupt:<br>
+ with repo.lock():<br>
</span>+ repo.svfs.rename('obsstore', 'obsstore.corrupt')<br>
<span class="">+ if len(data) > 0:<br>
+ repo.svfs.write('obsstore', data)<br>
+ ui.write(_('truncated damaged obsstore\n'))<br>
+ else:<br>
+ ui.write(_('deleted unreadable obsstore\n'))<br>
+ else:<br>
+ ui.write(_('no damage to obsstore\n'))<br>
+ else:<br>
+ ui.write(_('no obsstore\n'))<br>
+<br></span></blockquote><div><br></div><div>Please forgive me if this is pedantic, but I think this code should live in <a href="http://obsolete.pm">obsolete.pm</a>, where all the other code manipulating this file is located. Generally speaking, I think @command functions should only 1) check input/arguments 2) call into library function(s) 3) format output. Otherwise we end up with a commands.py that is even larger (and it is already grossly large IMO).<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
@command('debugwalk', walkopts, _('[OPTION]... [FILE]...'), inferrepo=True)<br>
def debugwalk(ui, repo, *pats, **opts):<br>
"""show how files match on given patterns"""<br>
</span>diff --git a/tests/test-completion.t b/tests/test-completion.t<br>
--- a/tests/test-completion.t<br>
+++ b/tests/test-completion.t<br>
@@ -109,6 +109,7 @@<br>
debugsub<br>
debugsuccessorssets<br>
debugtemplate<br>
+ debugtruncatestore<br>
debugwalk<br>
debugwireargs<br>
<br>
@@ -274,6 +275,7 @@<br>
debugsub: rev<br>
debugsuccessorssets:<br>
debugtemplate: rev, define<br>
+ debugtruncatestore: obsolete<br>
debugwalk: include, exclude<br>
debugwireargs: three, four, five, ssh, remotecmd, insecure<br>
files: rev, print0, include, exclude, template, subrepos<br>
<div><div class="h5">diff --git a/tests/test-debugcommands.t b/tests/test-debugcommands.t<br>
--- a/tests/test-debugcommands.t<br>
+++ b/tests/test-debugcommands.t<br>
@@ -126,3 +126,45 @@<br>
debugstacktrace.py:7 *in * (glob)<br>
debugstacktrace.py:6 *in g (glob)<br>
*/util.py:* in debugstacktrace (glob)<br>
+<br>
+Test corruption-fixing debugtruncatestore command<br>
+<br>
+ $ hg init corrupt-obsstore<br>
+ $ cd corrupt-obsstore<br>
+ $ cat >> .hg/hgrc << EOF<br>
+ > [experimental]<br>
+ > evolution = all<br>
+ > [extensions]<br>
+ > evolve =<br>
+ > EOF<br>
+ $ echo a > file<br>
+ $ hg commit -qAm file-a -d 1/1/2001<br>
+ $ hg debugobsolete<br>
+<br>
+ $ echo corrupt > .hg/store/obsstore<br>
+ $ hg debugobsolete 2> /dev/null<br>
+ [255]<br>
+ $ hg debugtruncatestore --obsolete<br>
</div></div>+ deleted unreadable obsstore<br>
<span class="">+ $ hg debugobsolete<br>
+<br>
+ $ echo bee > file<br>
+ $ hg commit -qAm file-b -d 1/1/2001<br>
+ $ echo b > file<br>
+ $ hg commit -qAm file-b -d 1/1/2001<br>
+ $ hg fold -m file-b -r '.^::.' -d 1/1/2001<br>
+ 2 changesets folded<br>
+ 1 files updated, 0 files merged, 0 files removed, 0 files unresolved<br>
+<br>
+ $ hg debugobsolete<br>
+ b429646ef7b48810f0aeaceb257eb589ff2e7e07 a7a6f057a0dda49d17b8ddb53d228494f939fed3 0 \(.*\) {'user': 'test'} (re)<br>
+ 12af3ef7db84073f5b5802cb875af46c6fc8f420 a7a6f057a0dda49d17b8ddb53d228494f939fed3 0 \(.*\) {'user': 'test'} (re)<br>
+ $ python -c 'print("CORRUPT" * 120)' >> .hg/store/obsstore<br>
+ $ hg debugobsolete 2> /dev/null<br>
+ [1]<br>
+ $ hg debugtruncatestore --obsolete<br>
</span>+ truncated damaged obsstore<br>
<span class="">+ $ hg debugobsolete<br>
+ b429646ef7b48810f0aeaceb257eb589ff2e7e07 a7a6f057a0dda49d17b8ddb53d228494f939fed3 0 \(.*\) {'user': 'test'} (re)<br>
+ 12af3ef7db84073f5b5802cb875af46c6fc8f420 a7a6f057a0dda49d17b8ddb53d228494f939fed3 0 \(.*\) {'user': 'test'} (re)<br>
+ $ cd ..<br>
</span>diff --git a/tests/test-help.t b/tests/test-help.t<br>
--- a/tests/test-help.t<br>
+++ b/tests/test-help.t<br>
@@ -917,6 +917,8 @@<br>
show set of successors for revision<br>
debugtemplate<br>
parse and apply a template<br>
+ debugtruncatestore<br>
<span class="">+ Fix up repository corruption by truncating damaged files<br>
</span> debugwalk show how files match on given patterns<br>
debugwireargs<br>
(no help text available)<br>
<div class="HOEnZb"><div class="h5">_______________________________________________<br>
Mercurial-devel mailing list<br>
<a href="mailto:Mercurial-devel@mercurial-scm.org">Mercurial-devel@mercurial-scm.org</a><br>
<a href="https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel" rel="noreferrer" target="_blank">https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel</a><br>
</div></div></blockquote></div><br></div></div>