Things we ought to do to improve our packaging
Matt Mackall
mpm at selenic.com
Tue Aug 6 17:53:04 CDT 2013
On Tue, 2013-08-06 at 15:04 -0500, Steve Borho wrote:
>
>
>
> On Tue, Aug 6, 2013 at 2:47 PM, Matt Mackall <mpm at selenic.com> wrote:
> On Tue, 2013-08-06 at 14:15 -0500, Steve Borho wrote:
> >
> >
> >
> > On Tue, Aug 6, 2013 at 2:01 PM, Augie Fackler
> <raf at durin42.com> wrote:
> > On Tue, Aug 06, 2013 at 01:53:34PM -0500, Matt
> Mackall wrote:
> > > [this discussion needs to be cross-posted]
> > >
> > > We've got a few long-standing problems with our
> current
> > packaging:
> > >
> > > - lag official release
> > > - not automated
> > > - not reproduceable by third parties
> > > - not signed
> > > - lack nightly builds
> > > - notable missing platforms (RHEL)
> > > - not uniform in configuration
> > >
> > > Our attitude to date has been "well, we're not
> really in
> > charge of that,
> > > that's done unofficially by volunteers". I think
> we've
> > reached a point
> > > where this needs to start changing.
> > >
> > > In an ideal world, a release would work like this:
> > >
> > > - I tag the release
> > > - I push it
> > > - Automated builders kick off on a bunch of VMs or
> hosts
> > > - Builders run stock makefile targets or scripts
> in tarball
> > > - Packages get signed
> > > - Packages get uploaded to one or more locations,
> including
> > > mercurial.selenic.com
> > > - Download links get updated
> > >
> > > Importantly, this same process would also be used
> to produce
> > automated
> > > nightly builds for testing and bug reporting and
> discovering
> > packaging
> > > problems before release dates.
> > >
> > > The job of packagers would thus shift from
> "(manually?)
> > building and
> > > uploading binaries" to "making sure the committed
> build
> > rules are
> > > correct and up-to-date".
> > >
> > > I think this also needs to be inclusive of builds
> for
> > packaging systems
> > > (Debian, Ubuntu, RHEL, Centos, Solaris, Pypi,
> etc.). These
> > actually
> > > suffer from more or less the same problems even
> though the
> > distribution
> > > model is different. So we should be building
> nightlies and
> > release
> > > builds for these systems and be more directly
> responsible
> > for quality
> > > and timeliness here as well.
> >
> >
> > This honestly sounds like a great improvement, given
> how stale
> > the
> > packages for Ubuntu tend to be, and how some
> packagers took it
> > upon
> > themselves in the past to enable all the extensions.
> >
> >
> >
> >
> > On the Windows side, everything is automated except for two
> steps:
> >
> > 1 - signing packages requires a passphrase to be entered
> > 2 - uploading is currently manually done through BB's clunky
> web
> > interface
> >
> >
> > If we had an SCP or FTP URL to upload to, and a sane policy
> for
> > deleting old nightlies after a short while, it could be
> automated.
>
>
> Well, you already have SCP access, no?
>
>
> Yep, it's just a question of whether you want nightlies in the same
> location as releases, or in a different folder that was size-managed.
>
> In addition to having a fully automated process that builds
> releases and
> nightlies, I think its also important to get to the build
> script into
> the tree so that people who need to build their own copies for
> various
> reasons (need bugfix, need custom deployment, St. Louis
> trampled by
> kaiju) can do so.
>
>
> So.. building windows frozen packages is fairly convoluted (especially
> for thg but the simple hg MSI is no cakewalk) and has a boatload of
> prereqs to be installed.
Yep, I'm well aware. And this is actually MORE reason to insist on full
automation. Let me back up and explain some of my motivation here:
While I think you do a great job with packaging, it's lately become
clear that package maintainers are a significant single point of
failure. And we've also not coincidentally discovered that we don't
actually know how some of our packages are actually built (like OS X)
because the instructions don't work any more. So when the packager
disappears, we've got several problems: find someone with the hardware,
install the OS(es), install the deps, reverse-engineer the build
process, hit unknown number of gotchas..
The OS X situation is not as complex as Windows, but it's still bad
enough that the current approach is clearly not good enough.
And that's not the only problem in this area. For instance, we also have
real uses for continuous builds that are not being met by monthly
releases. I really want the BTS robot to point people at builds and say
"try this and report back, please", but can't.
(And it's also a perennial confusion that we announce releases before
there are any packages built.)
Fortunately, we know how to fix these sorts of things:
- manifest the instructions and dependencies as build scripts
- version control the build scripts in the canonical repo
- use the build scripts, continuously and non-interactively
- get people to use those builds and discover problems before releases
- fix the problems found via check-in
With my 'hackable' Windows builds, I've gone as far as scripting the
download and install of the right versions of Python and gcc and all the
other deps if not present to make sure its reproduceable, so I know this
sort of thing can be done, even on Windows. I've obviously not yet gone
as far as putting this knowledge in the repo, but I ought to.
All that said, let's focus on phase 1: publishing nightly builds,
somehow, somewhere.
--
Mathematics is the supreme nostalgia of our time.
More information about the Mercurial-devel
mailing list