Cloning locally a repo with subrepos using ../ in .hgsub

Mike Williams obsd1 at eandem.co.uk
Sat Jan 29 08:48:37 CST 2011


On 01/29/11 00:10, Mads Kiilerich wrote:
> Mike Williams wrote, On 01/28/2011 04:23 PM:
>> Hi
>>
>> I am investigating the use of sub-repos to share different subsets of
>> a collection of libraries amongst many projects.  This approach has
>> been mentioned in other mailings to this list so I don't believe it is
>> an oddball requirement.
>>
>> So, I have created repos of the libraries at the same level as the
>> projects on the server, and then in each project reference the
>> sub-repos for the libraries in the .hgsub files as:
>>
>> libname = ../libname
>>
>> As expected this is all working fine when for creating clones from the
>> server.  The pain has started with creating local clones from the
>> clone created from the server - the relative paths no longer hold on
>> the local machine.  The aim is to follow the typical hg workflow of
>> having a clean clones to exchange changesets with the master repos,
>> and local clones to work on.
>>
>> Adding the subpaths extension and with a minimal .hg/subpaths file
>> containing(*)
>>
>> [subpaths]
>> ../(.*) = \1
>>
>> I can then clone locally from the master repo clone and everything
>> looks good until I want to interact with the master repos from the
>> original clone.  The simple solution is to rename/move .hg/subpaths
>> somewhere else for the duration of the interactions and then move it
>> back when working locally.
>>
>> There are two issues here I would like suggestions about:
>>
>> 1) Is there a better parent/sub-repo structure to represent this many
>> projects/many libraries structure on the server?  We have 10s of
>> projects and 100s of libraries with no reasonable project to "own" a
>> set of the libraries such that the normally recommended .hgsub entry
>> of libname = libname becomes usable.
>>
>> 2) There does seem to be an idiom here to convert a server's flat
>> sub-repo structure to a hierarchy locally.  Should this be rolled into
>> mercurial of the subpath extension, possibly just to remove leading
>> ../ from .hgsub entries?  This would be global command option
>> (--internal?) so it doesn't change any existing behaviour.
>>
>> Yes there is a downside with this in that no one is reproducing the
>> repo structure of the server, which makes rebuilding it after an
>> outage an interesting activity in its own right - perhaps a server
>> repo with all the repos being served as sub-repos of it.  But this is
>> disappearing up another line of thought so I will stop now.
>
> In my opinion subrepos are quite usable and stable, but not fully
> feature complete and best practice haven't been established yet. So you
> are right, the right way to solve  your problem isn't obvious.
>
> Much of what you mention and what could be improved in Mercurial was
> also "discussed" in the tread at
> http://selenic.com/pipermail/mercurial-devel/2011-January/027285.html .

I do remember seeing that, but I wanted to see how things could be made 
to work with what is currently available via subpaths.  I got close, but 
no cigar.

> With the risk repeating what I said in that thread:
>
> I think subrepos always should be used with trivial mappings as libname
> = libname. You have just given a good description of why other mappings
> is a bad idea. A consequence of trivial mappings is that repositories
> also must be stored "in place" in the tree structure on shared servers.
> You must thus either accept that the subrepos are stored many times on
> the server, or use symlinks or some kind of url rewrite to share the
> storage.

This would seem to be an issue with replicating a central server type 
workflow.  I can see it working better if you have a gatekeeper style 
workflow - someone is responsible from pushing back from the project's 
master repo sub-repo libraries to their specific master repos so other 
projects can pick up changes when they want to.  I'll have a go at 
setting that up as well to see how that flies.

> Note that Mercurial - also with subrepos - make working directory
> content 100% reproducible, so when a bug is fixed in a shared library it
> won't automagically show up in all repos that use it no matter what. You
> need to have a workflow for updating to the revision with the bugfix
> anyway, so the simple solution of storing it multiple times might not be
> that bad.

This will be one of the cultural changes that will be "interesting". 
Moving from an SCM where every project always picks up the tip of every 
library to one where every project has to be updated to pick up changes 
will meet some resistance (there has already been the question of how to 
find the set of projects using a library - the mental plan is to have a 
script to build a inverted list from the .hgsub file contents).

And yes there have been times where some project builds have been broken 
due to developer's not checking them, not to mention the contortions 
that have to be gone through to try and check across projects, 
especially on Windows even with junctions.  Changing many years of 
working practice will be fun.

>> (*) A little bit of experience from experimenting with the subpaths
>> extension.  The subpath is (currently) as it appears in .hgsub and not
>> normalised in anyway.  If you have mixture of Windows and unix path
>> separators present then you will need an entry of:
>>
>> ..(\\|/)(.*) = \2
>
> That sounds like a bug. Can you provide a full but minimal example that
> demonstrates the problem?

The following sequence for Windows will reproduce the issue.  This is 
with hg 1.7.3.

hg init subrepo1
echo H > subrepo1\file1
hg add subrepo1
hg commit -m "Initial sub-repo1 commit." subrepo1
hg init subrepo2
echo H > subrepo2\file1
hg add subrepo2
hg commit -m "Initial sub-repo2 commit." subrepo2
hg init repo
echo W > repo\file2
hg clone subrepo1 repo\subrepo1
hg clone subrepo2 repo\subrepo2
echo subrepo1 = ..\subrepo1 > repo\.hgsub
echo subrepo2 = ../subrepo2 >> repo\.hgsub
hg add repo
hg commit -m "Initial repo + sub-repos commit." repo
mkdir clone1-dir
hg clone repo clone1-dir\repo
echo [subpaths] > clone1-dir\repo\.hg\subpaths
echo ../(.*) = \1 >> clone1-dir\repo\.hg\subpaths
mkdir clone2-dir
hg clone clone1-dir\repo clone2-dir\repo

For me the final command produces the following output.2:

F:\temp\clonetest>hg clone clone1-dir\repo clone2-dir\repo
updating to branch default
pulling subrepo subrepo2 from F:\temp\clonetest\clone1-dir\repo\subrepo2
requesting all changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
pulling subrepo subrepo1 from F:\temp\clonetest\clone1-dir\repo\..\subrepo1
abort: repository F:\temp\clonetest\clone1-dir\repo\..\subrepo1 not found!

(Just spotted the "1 changesets with 1 changes to 1 files" which is 
jarring to read.  Is it policy not to bother with the singular/plural 
reporting issue?)

HTH

-- 
Mike


More information about the Mercurial mailing list