Many Projects/Repositories and common Libraries

[Chris Scott]

> In many cases I keep getting told 'you do that in the build process' BUT I do 
> not have a build process! I'm working on code in PHP, and Python and Ruby have 
> exactly the same problem. There is nothing to 'compile' and no target 'binary 
> build'. Packaging a distribution is now seen as simply taking a snapshot of some 
> repo, or 'just clone your own copy'. Which then fails because currently there is 
> no mechanism to include the bits that Chris is expecting to manage in the 
> 'artifact/dependency management system' ....

While you may not have a traditional compile step in your build process, I would argue that you probably still have one even if you don't call it as such.  "Build process" typically means the following for me:

1. Resolve dependencies. 
2. Compile 
3. Test
4. Dist (aka package for distribution; build documentation)
5. Publish

Even if you don't need to compile, or repackage for distribution, the concept of build tasks don't go away.  I believe that the chances approach 100% that organization with multiple projects will need a fully-featured build-chain eventually, so it makes sense to invest in it early.  It's a good bit a work to understand everything and set it up the way you want it to, but it's a time-saver in the long run.

It's perfectly reasonable to use hg for what you use it for, however.  I'm a big fan of KISS.

> IDEALLY what I need is to be able to define a 'super project' which has 

As Martin Geisler pointed out, hg subrepos seem to be a reasonable solution here(?) but I've not played with it.  My dependency chain is very-long, so it's nice to be able to build one project without having to re-build everything from source as one would presumably have to do with the sub-repo approach.  To do so would take hours for a simple build.  I still have the option to do so if I need to, however.

IMHO, once something becomes a dependency, it ceases being "source" and becomes "binary" (even for interpreted files that are distributed as-is), so I don't use a source control management tool, I use a dependency/artifact management tool.  This has a side effect of being able to mostly remove our need for having large binaries in hg; they're all in our ivy repo, which handles them much better.  All of our binaries are artifacts of build processes, not "sources" like images or audio, so obivously it wouldn't work for everybody.

> But the 
> main thing is to be able to manage links to such library projects such as 
> smarty, adodb and the other AJAX code bases. Traditionally these would be copies 
> held in a CVS or SVN master, but surely the whole point of DVCS is that we CAN 
> now maintain real links to the third party libraries, push bug fixes back to 
> those master copies, or maintain a linked branch that other users of the same 
> library can browse or clone. 

I like the idea of having the exact code repo for all of my dependencies, but many of my dependencies are closed source, do not have public access, or *gasp* do not build out-of-the-box.

A full build-chain (we use hg+ant+ivy+hudson) works for us in a platform and language agnostic way.  Now I have a process that works for both internal and external dependencies.  For the latter the process works for open and closed sources.  Is it overkill for some?  Maybe.  But I still think it's worth the investment.  Is it cumbersome to get started? Yep.  It could be a lot nicer. 

> All this is still very much 'work in progress' and with so many people branching off and doing their own thing to 
> meet their own requirements

That's true, in that there's no good "All-in-one" solution (outside of the Rational chain maybe?), but the pieces are well defined and play well with each other.  Major players, like Apache, have adopted a solution similar to what I've described, so I'm inclined to keep the "Render unto Caesar" philosophy: Let SCMs handle source code, which they do best; Let Build and dependency management handle what they do best.

~Chris