Bug 2556 - subrepo url normalization removes // when creating absolute ssh paths
Summary: subrepo url normalization removes // when creating absolute ssh paths
Status: RESOLVED FIXED
Alias: None
Product: Mercurial
Classification: Unclassified
Component: Mercurial (show other bugs)
Version: unspecified
Hardware: All All
: normal bug
Assignee: Bugzilla
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-12-15 16:12 UTC by Kevin Bullock
Modified: 2012-05-13 04:56 UTC (History)
7 users (show)

See Also:
Python Version: ---


Attachments
(34 bytes, application/x-sh)
2010-12-15 16:16 UTC, Kevin Bullock
Details
(34 bytes, application/x-sh)
2010-12-15 16:27 UTC, Kevin Bullock
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Kevin Bullock 2010-12-15 16:12 UTC
I generally use remote URLs of the form ssh://host//absolute/repo/path (note 
second double-slash) for internal projects, and I've started using subrepos 
regularly as well. Using a default path of that form with relative subrepo 
paths ('sub = sub' in .hgsub), `hg out -S` fails on checking the subrepo 
with:

remote: abort: There is no Mercurial repository here (.hg not found)!
abort: no suitable response from remote hg!

even though the repo is there.

Test script attached. Running with --debug (set DEBUG=--debug for test 
script) shows that with an absolute remote path, the subrepo path is 
computed wrongly:

running ssh localhost "hg -R Users/kbullock/Source/Explorations/hg/repo-
subrepo-ssh-abspath-clone/sub serve --stdio"

With a relative remote path (URL of the form ssh://host/relative/repo/path), 
the failure does not occur.
Comment 1 Kevin Bullock 2010-12-15 16:16 UTC
Coulda swore I attached that test script.
Comment 2 Kevin Bullock 2010-12-15 16:27 UTC
Correcting test script to make DEBUG=--debug actually work.
Comment 3 kiilerix 2010-12-15 17:27 UTC
Please post your output of your script.

Please also post the full output of the "hg out -S" you say fails.

You give an example of a path that is wrong. What should it have been?
Comment 4 Kevin Bullock 2010-12-15 19:56 UTC
My output from the script:

★  ./test-subrepo-ssh-abspath.sh 
adding a
adding .hgsub
committing subrepository sub

--- ABSOLUTE REMOTE PATH ---
0:baa6f37d2902
abort: default path for subrepository sub not found

comparing with ssh://localhost//Users/kbullock/Source/Explorations/hg/repo-
subrepo-ssh-abspath-clone
searching for changes
0:baa6f37d2902
remote: abort: There is no Mercurial repository here (.hg not found)!
abort: no suitable response from remote hg!
--> [failure here ^^^]

--- RELATIVE REMOTE PATH ---
0:baa6f37d2902
abort: default path for subrepository sub not found

comparing with ssh://localhost/Source/Explorations/hg/repo-subrepo-ssh-
abspath-clone
searching for changes
0:baa6f37d2902
comparing with ssh://localhost/Source/Explorations/hg/repo-subrepo-ssh-
abspath-clone/sub
searching for changes
0:fc0fbcbb375d


The given line from the debug output:

running ssh localhost "hg -R Users/kbullock/Source/Explorations/hg/repo-
subrepo-ssh-abspath-clone/sub serve --stdio"

_should_ list a path with a leading /, but as you can see it's trying to use 
a path relative to $HOME.
Comment 5 kiilerix 2010-12-16 05:29 UTC
Thank you. 

On 12/16/2010 02:56 AM, Kevin Bullock wrote:
> comparing with ssh://localhost//Users/kbullock/Source/Explorations/hg/repo-
> subrepo-ssh-abspath-clone
> searching for changes
> 0:baa6f37d2902
> remote: abort: There is no Mercurial repository here (.hg not found)!
> abort: no suitable response from remote hg!

Adding -v here will probably reveal that it uses the url 
ssh://localhost/Users/kbullock/... - that seems to be the crucial point.

That clarifies that it probably is the same issue as discussed in
http://www.selenic.com/pipermail/mercurial-devel/2010-September/024401.html

There has been some progress on a proper fix, but that endsup as a huge
cleanup - see 
https://bitbucket.org/brodie/mercurial-crew-mq/src/7e87205b1cbf/url-subrepo
- but so far this is a known issue.

The workaround so far is to use relative ssh urls or run "ln -s /Users
~kbullock".
Comment 6 Kevin Bullock 2010-12-16 10:18 UTC
Thanks for looking at this, Mads. Changing the title to reflect broader scope 
than `hg out -S`.
Comment 7 kiilerix 2010-12-16 11:00 UTC
Updating title further to describe the issue from a development perspective.

(Almost the same issue also popped up in the 2245fcd0e160 saga.)
Comment 8 Thomas Arendsen Hein 2011-03-31 04:11 UTC
just encountered this, too
Comment 9 Thomas Arendsen Hein 2011-03-31 04:18 UTC
... with Mercurial 1.8.1 and 5c18a0bca26f (current stable tip)
In 58b86b9149f1 (current default tip) this is solved, so marking as "testing".

Should the fix be ported to stable, too, or is the chance to break things
along the way too high?
Comment 10 kiilerix 2011-03-31 04:52 UTC
Yes, it was fixed by ce6227306c9a but we still need a test for it in the
test suite.

I think it would be safe to backport 4e8f2310f310 + ce6227306c9a and perhaps
974490c1768f + 58b86b9149f1 .
Comment 11 brodie 2011-03-31 11:44 UTC
Please add a test. I wasn't able to completely wrap my head around this issue to reproduce 
it.

If you backport any url patches, please also backport the changesets that correct the url 
object's parsing behavior and method/function names. I think that means you'd want to 
backport these changes:

changeset:   13770:4e8f2310f310
user:        Brodie Rao <brodie@bitheap.org>
date:        Fri Mar 25 22:58:56 2011 -0700
summary:     url: provide url object

changeset:   13771:ce6227306c9a
user:        Brodie Rao <brodie@bitheap.org>
date:        Fri Mar 25 22:59:04 2011 -0700
summary:     subrepos: use url.url when normalizing repo paths

changeset:   13814:03dfe0c85c1a
user:        Brodie Rao <brodie@bitheap.org>
date:        Wed Mar 30 19:50:56 2011 -0700
summary:     url: move drive letter checking into has_drive_letter() for extensions

changeset:   13815:d066d8d652c8
user:        Brodie Rao <brodie@bitheap.org>
date:        Wed Mar 30 20:00:23 2011 -0700
summary:     url: add trailing slashes to URLs with hostnames that don't have one

changeset:   13816:2540f8087e02
user:        Brodie Rao <brodie@bitheap.org>
date:        Wed Mar 30 20:00:24 2011 -0700
summary:     url: special case bundle URL parsing to preserve backwards compatibility

It'd also be nice to port the relevant bits from this changeset:

changeset:   13827:f1823b9f073b
tag:         tip
user:        Matt Mackall <mpm@selenic.com>
date:        Thu Mar 31 10:43:53 2011 -0500
summary:     url: nuke some newly-introduced underbars in identifiers
Comment 12 Thomas Arendsen Hein 2011-03-31 13:01 UTC
* Brodie Rao <bugs@mercurial.selenic.com> [20110331 19:44]:
> Please add a test. I wasn't able to completely wrap my head around this issue to reproduce 
> it.

With Python 2.5.2 (Debian lenny):
hg clone http://hg.intevation.org/mapwoc/
hg clone ssh://localhost/`pwd`/mapwoc/ mapwoc2

I can't reproduce it with Python 2.6.6 (Debian squeeze).

Which Python version do you use?

Regards,
Thomas
Comment 13 Thomas Arendsen Hein 2011-04-06 03:32 UTC
I tried applying the patches brodie listed and there are some conflicts that
are easily resolvable, except for f1823b9f073b which nearly completely
consists of rejected hunks.

But since the feature is broken for a long time (in Mercurial 1.6.x even on
python2.6) and no fresh regression, and the changes are more intrusive than
expected, I'm reluctant to having them in stable, but I noticed that there
is another problem in URL handling which only affects python 2.4/python2.5:
issue2754 (password of ssh URLs not replaced with ***)

So if the URL handling code has to be adjusted anyway, solving both problems
together in stable might be good.
Comment 14 Thomas Arendsen Hein 2011-04-07 04:37 UTC
Should I push this minimal workaround to stable?

All tests run fine with python2.5 on Debian lenny and with python2.6 on
Debian squeeze.

# HG changeset patch
# Parent 08d49b6b8d320eb7cf6705a6eefe68c25a39bdba
# User Thomas Arendsen Hein <thomas@intevation.de>
# Date 1302172427 -7200
subrepo: prevent url normalization from removing // in ssh paths (issue2556)

diff -r 08d49b6b8d32 mercurial/subrepo.py
--- a/mercurial/subrepo.py	Thu Apr 07 13:23:07 2011 +0530
+++ b/mercurial/subrepo.py	Thu Apr 07 12:33:48 2011 +0200
@@ -202,8 +202,12 @@
                 if parent[-1] == '/':
                     parent = parent[:-1]
                 r = urlparse.urlparse(parent + '/' + source)
-                r = urlparse.urlunparse((r[0], r[1],
-                                         posixpath.normpath(r[2]),
+                if parent.startswith('ssh://'):
+                    host, path = r[2][2:].split('/', 1)
+                    r2 = '//%s/%s' % (host, posixpath.normpath(path))
+                else:
+                    r2 = posixpath.normpath(r[2])
+                r = urlparse.urlunparse((r[0], r[1], r2,
                                          r[3], r[4], r[5]))
                 return r
             else: # plain file system path
Comment 15 Matt Mackall 2011-04-07 10:33 UTC
On the list, please.
Comment 16 Thomas Arendsen Hein 2011-04-07 10:49 UTC
Sent to the list:
http://thread.gmane.org/gmane.comp.version-control.mercurial.devel/39944
Comment 17 HG Bot 2011-04-07 15:00 UTC
Fixed by http://selenic.com/repo/hg/rev/71ea5b2b9517
Thomas Arendsen Hein <thomas@intevation.de>
subrepo: prevent url normalization from removing // in ssh paths (issue2556)

(please test the fix)
Comment 18 Thomas Arendsen Hein 2011-04-07 15:21 UTC
works for me in all combinations (python 2.5 and 2.6 with stable and
default), so setting to resolved.
Comment 19 Bugzilla 2012-05-12 09:15 UTC

--- Bug imported by bugzilla@serpentine.com 2012-05-12 09:15 EDT  ---

This bug was previously known as _bug_ 2556 at http://mercurial.selenic.com/bts/issue2556
Imported an attachment (id=1495)
Imported an attachment (id=1496)