Bug 3528 - Support conversion of git submodules
Summary: Support conversion of git submodules
Status: RESOLVED ARCHIVED
Alias: None
Product: Mercurial
Classification: Unclassified
Component: convert (show other bugs)
Version: earlier
Hardware: All All
: normal feature
Assignee: Bugzilla
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-07-03 14:57 UTC by cowwoc2020
Modified: 2017-11-01 18:05 UTC (History)
6 users (show)

See Also:
Python Version: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description cowwoc2020 2012-07-03 14:57 UTC
According to bug #2150, converting git submodules isn't supported yet.

Please provide a workaround (is there one?) and work to support this kind of conversion out-of-the-box.
Comment 1 cowwoc2020 2012-07-03 15:31 UTC
I just realized that the repository I am trying to convert is related to the one Matt was complaining about in bug #2150 :) Specifically, I am trying to convert git://anongit.freedesktop.org/gstreamer/gstreamer

All their repositories seem to link against "gstreamer/common".

I'd like to fork their repositories, convert to hg, and occasionally provide patches (pulling from upstream on a regular basis and doing an incremental convert). I'd like to avoid hosting my own git repository if possible. What can I do?
Comment 2 cowwoc2020 2012-07-06 10:03 UTC
Perhaps we can introduce an option (similar to "convert.hg.ignoreerrors") for skipping git submodules as a workaround? In the case of GStreamer, I don't really need the submodule information to work in a private branch and contribute my changes back.
Comment 3 HG Bot 2012-11-15 17:15 UTC
Fixed by http://selenic.com/repo/hg/rev/0eed66327ad4
YaNan Xu <robot9@fb.com>
convert: add support for converting git submodule (issue3528)

Previously, convert aborted upon encountering a git submodule. This patch
changes it so that it now succeeds. It modifies convert_git to manually generate
'.hgsub' and '.hgsubstate' files for each git revision, so as to convert git sub
modules to non-mercurial subrepositories.

(please test the fix)
Comment 4 cowwoc2020 2012-11-18 00:19 UTC
Yanan,

Converting sub-modules still fails.

Testcase:

1. hg clone http://selenic.com/repo/hg
2. hg update 0eed66327ad4
3. Added "conv=C:/Users/Gili/Documents/convert/hg/hgext/convert" into mercurial.ini
4. git clone git://anongit.freedesktop.org/gstreamer/gstreamer
5. hg conv gstreamer
6. Operation fails with: "hg: parse error at .gitmodules:2:         path = common"
Comment 5 YaNan Xu 2012-12-18 15:02 UTC
Hi Gili,
Sorry to reply so late... I haven't got any message about this

There seems some problem while parsing the .gitmodules file, I firstly manually parse it but then changed to use mercurial.config.config().parse()

the .gitmodules file is like this:
[submodule "common"]
        path = common
        url = git://anongit.freedesktop.org/gstreamer/common

The problem is there are tabs(or spaces) start at each item of a section and I just deal with tabs. The mercurial.config.config().parse() can not parse items starts with spaces or tabs, so I did a replace to remove tabs before parsing, which is good in my computer. But I didn't handle the situation that spaces come first.

There are two solutions come:
One is to delete prepend spaces in the converter code
ANother is do some change to mercurial.config.config().parse() so that it can support items start with tabs or spaces.

But I think the first one should be a little easier :)
Comment 6 YaNan Xu 2012-12-18 20:47 UTC
(In reply to comment #4)

I have changed the code and tested with your git project, and it took 3 hours to finish the convert, and the result turns to be good to me :)

Notice: you can go to the new converted dir and run "hg up", but the "common" folder may not appear, just co to one previous changeset and co back: "hg co 15651:9571e5fe8461" then "hg co master", the submodule will appear...

I don't why and I'll spend sometime to look for it. Hope it works for you!

Here is the patch and you may use it temporarily, Thanks! 

# HG changeset patch
# User YaNan Xu <robot9@fb.com>
# Date 1355863261 28800
# Node ID 3c3bee00b600e592fc9be492c1184e81958b6716
# Parent  b2779b01d1296afab1e78b8d1d714adfc67441b1
convert: parsing git submodule config file (issue3528)

Add support for parsing git submodule file in which items start with spaces

diff --git a/hgext/convert/git.py b/hgext/convert/git.py
--- a/hgext/convert/git.py
+++ b/hgext/convert/git.py
@@ -5,7 +5,7 @@
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.

-import os
+import os, re
 from mercurial import util, config
 from mercurial.node import hex, nullid
 from mercurial.i18n import _
@@ -109,13 +109,14 @@
     def parsegitmodules(self, content):
         """Parse the formatted .gitmodules file, example file format:
         [submodule "sub"]\n
-        \tpath = sub\n
-        \turl = git://giturl\n
+        [\t ]*path = sub\n
+        [\t ]*url = git://giturl\n
         """
         self.submodules = []
         c = config.config()
-        # Each item in .gitmodules starts with \t that cant be parsed
-        c.parse('.gitmodules', content.replace('\t',''))
+        # Each item in .gitmodules starts with spaces or \t that cant be parsed
+        content = re.sub(r'[\t ]+(\S+\s*=)', r'\1', content)
+        c.parse('.gitmodules', content)
         for sec in c.sections():
             s = c[sec]
             if 'url' in s and 'path' in s:
Comment 7 YaNan Xu 2012-12-19 01:45 UTC
Sorry...I just found the branch "default" of the converted hg repos isn't master, just "hg co master" should work. thanks!
Comment 8 cowwoc2020 2013-01-14 11:38 UTC
YaNan,

How do I apply your changeset for testing?

I tried checking out HEAD and running "hg import convert.patch" and got:

applying convert.patch
patching file hgext/convert/git.py
Hunk #1 FAILED at 4
Hunk #2 FAILED at 108
2 out of 2 hunks FAILED -- saving rejects to file hgext/convert/git.py.rej
abort: patch failed to apply

Alternatively, you could commit your fix and I could test it that way.
Comment 9 cowwoc2020 2013-06-17 14:54 UTC
Changing the status from "in progress" to "confirmed" because I haven't received a reply in over 6 months. This issue needs a new owner.
Comment 10 Jeff 2014-03-19 11:15 UTC
I had the same problem recently with leading spaces in a .gitmodules file during conversion:

hg: parse error at .gitmodules:2:     path = cim

Applying the patch to hgext/convert/git.py solved that problem, thanks!

Now, however, the conversion fails to parse the submodules due to the .gitmodules not existing:

fatal: Path '.gitmodules' does not exist in 'a99d390d4ccd6ffb66a245e89d2223df1f56fde4'
abort: cannot read submodules config file in a99d390d4ccd6ffb66a245e89d2223df1f56fde4

The logic makes sense, but it seems this situation should be handled more gracefully.
Comment 11 Bugzilla 2015-02-27 01:03 UTC
Bug was inactive for 344 days, archiving