D3759: packaging: replace dockerlib.sh with a Python script

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  I want to do some more advanced things with Docker in upcoming
  commits. Trying to do that with shell scripts will be a bit too
  painful for my liking. Implementing things in Python will be
  vastly simpler in the long run.
  
  This commit essentially ports dockerlib.sh to a Python script.
  dockerdeb and dockerrpm have been ported to use the new hg-docker
  script.
  
  hg-docker requires Python 3. I've only tested on Python 3.5.
  
  Unlike the local packaging scripts which may need to run on old
  distros, the Docker packaging scripts don't have these constraints.
  So I think it is acceptable to require Python 3.5.
  
  As part of the transition, the Docker image tags changed slightly.
  I don't think that's a big deal: the Docker image names are
  effectively arbitrary and are a means to an end to achieve
  running commands in Docker containers.
  
  The code for resolving the Dockerfile content allows substituting
  values passed as arguments. This will be used in a subsequent commit.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3759

AFFECTED FILES
  contrib/packaging/dockerdeb
  contrib/packaging/dockerlib.sh
  contrib/packaging/dockerrpm
  contrib/packaging/hg-docker

CHANGE DETAILS

diff --git a/contrib/packaging/hg-docker b/contrib/packaging/hg-docker
new file mode 100755
--- /dev/null
+++ b/contrib/packaging/hg-docker
@@ -0,0 +1,118 @@
+#!/usr/bin/env python3
+#
+# Copyright 2018 Gregory Szorc <gregory.szorc at gmail.com>
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+
+import argparse
+import pathlib
+import shutil
+import subprocess
+import sys
+
+
+def get_docker() -> str:
+    docker = shutil.which('docker.io') or shutil.which('docker')
+    if not docker:
+        print('could not find docker executable')
+        return 1
+
+    try:
+        out = subprocess.check_output([docker, '-h'], stderr=subprocess.STDOUT)
+
+        if b'Jansens' in out:
+            print('%s is the Docking System Tray; try installing docker.io' %
+                  docker)
+            sys.exit(1)
+    except subprocess.CalledProcessError as e:
+        print('error calling `%s -h`: %s' % (docker, e.output))
+        sys.exit(1)
+
+    out = subprocess.check_output([docker, 'version'],
+                                  stderr=subprocess.STDOUT)
+
+    lines = out.splitlines()
+    if not any(l.startswith((b'Client:', b'Client version:')) for l in lines):
+        print('`%s version` does not look like Docker' % docker)
+        sys.exit(1)
+
+    if not any(l.startswith((b'Server:', b'Server version:')) for l in lines):
+        print('`%s version` does not look like Docker' % docker)
+        sys.exit(1)
+
+    return docker
+
+
+def get_dockerfile(path: pathlib.Path, args: list) -> bytes:
+    with path.open('rb') as fh:
+        df = fh.read()
+
+    for k, v in args:
+        df = df.replace(b'%%%s%%' % k, v)
+
+    return df
+
+
+def build_docker_image(dockerfile: pathlib.Path, params: list, tag: str):
+    """Build a Docker image from a templatized Dockerfile."""
+    docker = get_docker()
+
+    dockerfile_path = pathlib.Path(dockerfile)
+
+    dockerfile = get_dockerfile(dockerfile_path, params)
+
+    print('building Dockerfile:')
+    print(dockerfile.decode('utf-8', 'replace'))
+
+    args = [
+        docker,
+        'build',
+        '--build-arg', 'http_proxy',
+        '--build-arg', 'https_proxy',
+        '--tag', tag,
+        '-',
+    ]
+
+    print('executing: %r' % args)
+    subprocess.run(args, input=dockerfile, check=True)
+
+
+def command_build(args):
+    build_args = []
+    for arg in args.build_arg:
+        k, v = arg.split('=', 1)
+        build_args.append((k.encode('utf-8'), v.encode('utf-8')))
+
+    build_docker_image(pathlib.Path(args.dockerfile),
+                       build_args,
+                       args.tag)
+
+
+def command_docker(args):
+    print(get_docker())
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+
+    subparsers = parser.add_subparsers(title='subcommands')
+
+    build = subparsers.add_parser('build', help='Build a Docker image')
+    build.set_defaults(func=command_build)
+    build.add_argument('--build-arg', action='append', default=[],
+                        help='Substitution to perform in Dockerfile; '
+                             'format: key=value')
+    build.add_argument('dockerfile', help='path to Dockerfile to use')
+    build.add_argument('tag', help='Tag to apply to created image')
+
+    docker = subparsers.add_parser('docker-path', help='Resolve path to Docker')
+    docker.set_defaults(func=command_docker)
+
+    args = parser.parse_args()
+
+    return args.func(args)
+
+
+if __name__ == '__main__':
+    sys.exit(main())
diff --git a/contrib/packaging/dockerrpm b/contrib/packaging/dockerrpm
--- a/contrib/packaging/dockerrpm
+++ b/contrib/packaging/dockerrpm
@@ -1,16 +1,16 @@
 #!/bin/bash -e
 
-. $(dirname $0)/dockerlib.sh
-
 BUILDDIR=$(dirname $0)
 export ROOTDIR=$(cd $BUILDDIR/../..; pwd)
 
-checkdocker
-
 PLATFORM="$1"
 shift # extra params are passed to buildrpm
 
-initcontainer $PLATFORM
+DOCKER=$($BUILDDIR/hg-docker docker-path)
+
+CONTAINER=hg-docker-$PLATFORM
+
+$BUILDDIR/hg-docker build $BUILDDIR/docker/$PLATFORM $CONTAINER
 
 RPMBUILDDIR=$ROOTDIR/packages/$PLATFORM
 $ROOTDIR/contrib/packaging/buildrpm --rpmbuilddir $RPMBUILDDIR --prepare $*
diff --git a/contrib/packaging/dockerlib.sh b/contrib/packaging/dockerlib.sh
deleted file mode 100644
--- a/contrib/packaging/dockerlib.sh
+++ /dev/null
@@ -1,30 +0,0 @@
-#!/bin/sh -eu
-
-# This function exists to set up the DOCKER variable and verify that
-# it's the binary we expect. It also verifies that the docker service
-# is running on the system and we can talk to it.
-function checkdocker() {
-  if which docker.io >> /dev/null 2>&1 ; then
-    DOCKER=docker.io
-  elif which docker >> /dev/null 2>&1 ; then
-    DOCKER=docker
-  else
-    echo "Error: docker must be installed"
-    exit 1
-  fi
-
-  $DOCKER -h 2> /dev/null | grep -q Jansens && { echo "Error: $DOCKER is the Docking System Tray - install docker.io instead"; exit 1; }
-  $DOCKER version | grep -Eq "^Client( version)?:" || { echo "Error: unexpected output from \"$DOCKER version\""; exit 1; }
-  $DOCKER version | grep -Eq "^Server( version)?:" || { echo "Error: could not get docker server version - check it is running and your permissions"; exit 1; }
-}
-
-# Construct a container and leave its name in $CONTAINER for future use.
-function initcontainer() {
-  [ "$1" ] || { echo "Error: platform name must be specified"; exit 1; }
-
-  DFILE="$ROOTDIR/contrib/packaging/docker/$1"
-  [ -f "$DFILE" ] || { echo "Error: docker file $DFILE not found"; exit 1; }
-
-  CONTAINER="hg-dockerrpm-$1"
-  cat $DFILE | $DOCKER build --build-arg http_proxy --build-arg https_proxy --tag $CONTAINER -
-}
diff --git a/contrib/packaging/dockerdeb b/contrib/packaging/dockerdeb
--- a/contrib/packaging/dockerdeb
+++ b/contrib/packaging/dockerdeb
@@ -1,21 +1,21 @@
 #!/bin/bash -eu
 
-. $(dirname $0)/dockerlib.sh
 . $(dirname $0)/packagelib.sh
 
 BUILDDIR=$(dirname $0)
 export ROOTDIR=$(cd $BUILDDIR/../.. > /dev/null; pwd)
 
-checkdocker
-
 DISTID="$1"
 CODENAME="$2"
 PLATFORM="$1-$2"
 shift; shift # extra params are passed to build process
 
 OUTPUTDIR=${OUTPUTDIR:=$ROOTDIR/packages/$PLATFORM}
+CONTAINER=hg-docker-$PLATFORM
 
-initcontainer $PLATFORM
+DOCKER=$($BUILDDIR/hg-docker docker-path)
+
+$BUILDDIR/hg-docker build $BUILDDIR/docker/$PLATFORM $CONTAINER
 
 # debuild only appears to be able to save built debs etc to .., so we
 # have to share the .. of the current directory with the docker



To: indygreg, #hg-reviewers
Cc: mercurial-devel