Working docker file - tests having some errors but sandbox functionin…

…g- test publish workflow Signed-off-by: Peter Baker <[email protected]>
open-AIMS · Aug 29, 2024 · f264b8a · f264b8a
1 parent be62954
commit f264b8a
Show file tree

Hide file tree

Showing 4 changed files with 122 additions and 34 deletions.
diff --git a/.github/workflows/PublishDockerImage.yml b/.github/workflows/PublishDockerImage.yml
@@ -0,0 +1,77 @@
+# This workflow builds and publishes the ADRIA base Docker image to GitHub Container Registry (ghcr.io)
+# It is triggered when a new release is published in the repository
+
+# Additional notes:
+# - The workflow uses the github.repository context to name the image, ensuring it's tied to your repository
+# - The GITHUB_TOKEN is automatically provided by GitHub Actions, no need to set it up manually
+# - The Docker metadata action automatically generates appropriate tags based on the release version
+# - The Julia version can be easily updated by changing the JULIA_VERSION environment variable at the top of the workflow
+
+name: Build and Publish ADRIA Base Docker Image
+
+on:
+  release:
+    types: [published]
+  push:
+    branches:
+      - feat/merging-adria-docker
+
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: ${{ github.repository }}/adria-base
+  JULIA_VERSION: 1.10.4
+
+jobs:
+  build-and-push-image:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+
+    steps:
+      # Step 1: Checkout the repository code
+      - name: Checkout repository
+        uses: actions/checkout@v3
+
+      # Step 2: Extract the version number from the release tag
+      # This removes the 'v' prefix from the tag (e.g., 'v1.2.3' becomes '1.2.3')
+      - name: Extract version from tag
+        id: get_version
+        run: echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_OUTPUT
+
+      # Step 3: Log in to the GitHub Container Registry
+      # This uses the provided GitHub token for authentication
+      - name: Log in to the Container registry
+        uses: docker/login-action@v2
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      # Step 4: Extract metadata for Docker
+      # This step generates tags and labels for the Docker image
+      - name: Extract metadata (tags, labels) for Docker
+        id: meta
+        uses: docker/metadata-action@v4
+        with:
+          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+          tags: |
+            type=semver,pattern={{version}}
+            type=semver,pattern={{major}}.{{minor}}
+            type=semver,pattern={{major}}
+            latest
+
+      # Step 5: Build and push the Docker image
+      # This step builds the adria-base image and pushes it to the registry
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v4
+        with:
+          context: .
+          target: adria-base # Specifies which stage of the Dockerfile to build
+          push: true # Pushes the image to the registry
+          tags: ${{ steps.meta.outputs.tags }} # Uses the tags generated in the metadata step
+          labels: ${{ steps.meta.outputs.labels }} # Uses the labels generated in the metadata step
+          # Passes the Julia and ADRIA versions to the Dockerfile
+          build-args: |
+            ADRIA_VERSION=${{ steps.get_version.outputs.VERSION }}
+            JULIA_VERSION=${{ env.JULIA_VERSION }}
diff --git a/Dockerfile b/Dockerfile
@@ -1,13 +1,13 @@
+# What base image should be used for the sandbox
 ARG SANDBOX_FROM="adria-dev"
 
+# See https://hub.docker.com/_/julia for valid versions.
+ARG JULIA_VERSION="1.10.4"
+
 #------------------------------------------------------------------------------
 # internal-base build target: julia with OS updates and an empty @adria
 # Julia environment prepared for use. NOT intended for standalone use.
 #------------------------------------------------------------------------------
-# Work with an apt-based Debian 12 (bookworm) OS, but allow
-# the Julia platform version to be overridden at build-time
-# See https://hub.docker.com/_/julia for valid versions.
-ARG JULIA_VERSION="1.10.4"
 FROM julia:${JULIA_VERSION}-bookworm AS internal-base
 
 # Record the actual base image used from the FROM command as label in the compiled image
@@ -21,9 +21,9 @@ RUN --mount=target=/var/lib/apt/lists,type=cache,sharing=locked \
     apt-get update \
     && apt-get -y upgrade \
     && apt-get install --no-install-recommends -y \
-        git \
-        less \
-        nano \
+    git \
+    less \
+    nano \
     && apt-get clean \
     && apt-get autoremove --purge \
     && rm -rf /var/lib/apt/lists/*
@@ -35,7 +35,7 @@ RUN --mount=target=/var/lib/apt/lists,type=cache,sharing=locked \
 # user accounts, but still activate environments configured by this dockerfile.
 ENV JULIA_DEPOT_PATH="/usr/local/share/julia"
 
-# Prepare an empty @adria Julia environment for derived images to use
+# Prepare an empty @adria Julia environment for derived images to use - this is created in the shared depot path
 RUN mkdir -p "${JULIA_DEPOT_PATH}" && \
     chmod 0755 "${JULIA_DEPOT_PATH}" && \
     julia -e 'using Pkg; Pkg.activate("adria", shared=true)'
@@ -56,18 +56,23 @@ ENTRYPOINT ["julia"]
 #------------------------------------------------------------------------------
 FROM internal-base AS adria-base
 
+# What version of ADRIA from package registry to install in adria-base
+ARG ADRIA_VERSION="0.11.0"
+
+# Which julia package registry version to install
+ENV ADRIA_VERSION=$ADRIA_VERSION
+
+
 # Install ADRIA.jl into the @adria shared environment as an unregistered package.
 # - Allow the package source and version to be overridden at build-time.
 # - Include citation information for ADRIA.jl in the image labels.
-ARG ADRIA_REPO="https://github.com/open-AIMS/ADRIA.jl.git" \
-    ADRIA_REFSPEC="main"
 RUN mkdir -p "${JULIA_DEPOT_PATH}" && \
     chmod 0755 "${JULIA_DEPOT_PATH}" && \
-    julia --project=@adria -e "using Pkg; Pkg.add(url=\"${ADRIA_REPO}\", rev=\"${ADRIA_REFSPEC}\"); using ADRIA"
-LABEL au.gov.aims.adria.source="${ADRIA_REPO}" \
-      au.gov.aims.adria.branch="${ADRIA_REFSPEC}" \
-      au.gov.aims.adria.vendor="Australian Institute of Marine Science" \
-      au.gov.aims.adria.licenses=MIT
+    julia --project=@adria -e "using Pkg; Pkg.add(name=\"ADRIA\", version=\"${ADRIA_VERSION}\"); using ADRIA"
+LABEL au.gov.aims.adria.source="https://github.com/open-AIMS/ADRIA.jl/releases/tag/v${ADRIA_VERSION}" \
+    au.gov.aims.adria.version="${ADRIA_VERSION}" \
+    au.gov.aims.adria.vendor="Australian Institute of Marine Science" \
+    au.gov.aims.adria.licenses=MIT
 
 #------------------------------------------------------------------------------
 # adria-dev build target: Assumes you have the ADRIA.jl source code
@@ -91,17 +96,16 @@ ENV ADRIA_ENV_DIR="${JULIA_DEPOT_PATH}/environments/adria" \
 #       in this adria-docker repository instead is a horrible hack, but works.
 WORKDIR "${ADRIA_SRC_DIR}"
 COPY ./Project.toml ./Project.toml
-# TODO bring this back but for now let it build the manifest
-#COPY ./Manifest.toml ./Manifest.toml
-RUN julia --project=@adria -e 'using Pkg;  Pkg.instantiate(verbose=true);'
+COPY ./Manifest.toml ./Manifest.toml
+RUN julia --project=@adria -e 'using Pkg;  Pkg.instantiate(verbose=true)'
 
 # Install the ADRIA source code and configure it as a development
 # package in the @adria shared environment.
 # Should be v speedy if the .toml file is unchanged, because all the
 # dependencies *should* already be installed.
 COPY . .
 RUN julia --project=@adria \
-          -e  'using Pkg; Pkg.develop(PackageSpec(path=pwd())); Pkg.precompile(); using ADRIA;'
+    -e  'using Pkg; Pkg.develop(PackageSpec(path=pwd())); Pkg.precompile(); using ADRIA;'
 
 
 #------------------------------------------------------------------------------

diff --git a/README.md b/README.md
@@ -2,7 +2,7 @@
 
 ADRIA: Adaptive Dynamic Reef Intervention Algorithms.
 
-[![Release](https://img.shields.io/github/v/release/open-AIMS/ADRIA.jl)](https://github.com/open-AIMS/ADRIA.jl/releases)  [![DOI](https://zenodo.org/badge/483052659.svg)](https://zenodo.org/badge/latestdoi/483052659)
+[![Release](https://img.shields.io/github/v/release/open-AIMS/ADRIA.jl)](https://github.com/open-AIMS/ADRIA.jl/releases) [![DOI](https://zenodo.org/badge/483052659.svg)](https://zenodo.org/badge/latestdoi/483052659)
 
 [![Documentation](https://img.shields.io/badge/docs-stable-blue)](https://open-aims.github.io/ADRIA.jl/stable/) [![Documentation](https://img.shields.io/badge/docs-dev-blue)](https://open-aims.github.io/ADRIA.jl/dev/)
 
@@ -37,6 +37,7 @@ For developers, refer to the
 
 This repo contains a multistage Dockerfile for generating containerised applications based on the ADRIA source code in this repository.
 
+- [docker configuration](#docker-configuration)
 - [adria-base](#adria-base)
   - [Building `adria-base`](#building-adria-base)
   - [Running `adria-base` with a non-interactive Julia command](#running-adria-base-with-a-non-interactive-julia-command)
@@ -55,6 +56,16 @@ This repo contains a multistage Dockerfile for generating containerised applicat
 
 ---
 
+## Docker configuration
+
+The following build args and defaults are available to configure the build behaviour.
+
+- ARG SANDBOX_FROM="adria-dev": What base image should be used for the sandbox
+
+- ARG ADRIA_VERSION="0.11.0": What version of ADRIA from package registry to install in adria-base
+
+- ARG JULIA_VERSION="1.10.4": See https://hub.docker.com/\*/julia for valid versions.
+
 ## adria-base
 
 The `adria-base` image variant is a Julia image with [ADRIA.jl](https://github.com/open-AIMS/ADRIA.jl) installed to a shared environment that is included in the Julia LOAD_PATH directly from it's github origin.
@@ -75,9 +86,9 @@ docker build --target "adria-base" --tag ADRIA.jl/adria-base:latest .
 
 You can also opt to specify some custom [build arguments](https://docs.docker.com/reference/cli/docker/image/build/#build-arg) to change the versions of Julia or ADRIA.jl that get installed. Supported arguments are:
 
-- `ADRIA_REPO`:  URL for the repository that ADRIA.jl should be cloned from. Defaults to <https://github.com/open-AIMS/ADRIA.jl.git>
+- `ADRIA_REPO`: URL for the repository that ADRIA.jl should be cloned from. Defaults to <https://github.com/open-AIMS/ADRIA.jl.git>
 - `ADRIA_REFSPEC`: the branch-name or tag of the `ADRIA_REPO` that you want to install. Defaults to `main`.
-- `JULIA_VERSION`: The version of the Julia platform you want to install ADRIA.jl into.  This must be one of the versions available for the official [Julia base image](https://hub.docker.com/_/julia). Defaults to `1.10.1`.
+- `JULIA_VERSION`: The version of the Julia platform you want to install ADRIA.jl into. This must be one of the versions available for the official [Julia base image](https://hub.docker.com/_/julia). Defaults to `1.10.1`.
 
 See the [docker-compose.yaml](./docker-compose.yaml) file for an example of how to specify build arguments in docker compose.
 
@@ -87,7 +98,6 @@ See the [docker-compose.yaml](./docker-compose.yaml) file for an example of how
 
 e.g. to list the packages installed in the `@adria` shared environment:
 
-
 ```bash
 # EITHER using docker compose:
 docker compose run --rm adria-base --project=@adria -e 'using Pkg; Pkg.status()'
@@ -98,7 +108,6 @@ docker run --rm ADRIA.jl/adria-base:latest --project=@adria -e 'using Pkg; Pkg.s
 
 &nbsp;
 
-
 ### Running `adria-base` as an interactive Julia shell
 
 To launch an interactive Julia shell:
@@ -115,10 +124,9 @@ In both cases, type `CTRL-d` to exit the shell and stop the container.
 
 &nbsp;
 
-
 ### Running `adria-base` with a non-Julia entrypoint
 
-If you want to use this image to run something *other* than a Julia command, you can specify an alternate entrypoint at runtime as well as an alternate command. e.g. to launch an interactive `bash` shell in the container for checking filesystem permissions or similar:
+If you want to use this image to run something _other_ than a Julia command, you can specify an alternate entrypoint at runtime as well as an alternate command. e.g. to launch an interactive `bash` shell in the container for checking filesystem permissions or similar:
 
 ```bash
 # EITHER using docker compose:
@@ -145,11 +153,11 @@ The section of the [Dockerfile](./Dockerfile) that defines the `adria-sandbox` t
 
 ## adria-dev
 
-The `adria-dev` image variant is an *alternative* to the `adria-base` image, not a derived application.
+The `adria-dev` image variant is an _alternative_ to the `adria-base` image, not a derived application.
 Instead of installing [ADRIA.jl](https://github.com/open-AIMS/ADRIA.jl) as a normal package, it looks for
 the ADRIA.jl source code in a local subdirectory, and installs that as a [Julia development package](https://pkgdocs.julialang.org/v1/api/#Pkg.develop).
 
-This allows you to use the `adria-dev` container as an `ADRIA.jl` development environment:  you can run tests,
+This allows you to use the `adria-dev` container as an `ADRIA.jl` development environment: you can run tests,
 bind-mount and edit the code, re-resolve dependencies and all sorts of other useful things without needing
 to a native installation of Julia.
 
@@ -164,7 +172,7 @@ If you are not familiar with git submodules, some useful commands are:
 - to switch to an existing branch of ADRIA.jl: `cd ADRIA.jl && git checkout <branch-name>`
 - to pull recent commits on your current branch of ADRIA.jl: `cd ADRIA.jl && git pull`
 - To commit and push changes you have made to ADRIA.jl source:
-  - First, commit and push just as usual from *inside* the submodule repository. As per good development practice, work in a dedicated branch:
+  - First, commit and push just as usual from _inside_ the submodule repository. As per good development practice, work in a dedicated branch:
     ```bash
     cd ADRIA.jl
     git checkout -b my-dev-branch
@@ -174,7 +182,7 @@ If you are not familiar with git submodules, some useful commands are:
     ```
   - At this point, your changes will be safely pushed to a new branch of the
     [ADRIA.jl origin repository](https://github.com/open-AIMS/ADRIA.jl), and you can make pull requests just like normal.
-  - Then from the root of *this* repository:
+  - Then from the root of _this_ repository:
     ```bash
     git add ADRIA.jl
     git commit -m"explain your changes again"
@@ -198,7 +206,6 @@ The same `JULIA_VERSION` build argument that works with `adria-base` will also w
 
 &nbsp;
 
-
 ### Running `adria-dev` as an interactive Julia shell
 
 Very useful for running commands to update Package manifests and similar!
@@ -245,7 +252,7 @@ This method of running tests is suitable to use in a containerised continuous in
 
 The `adria-sandbox` image variant is set up to run the `sandbox` Julia application which has its source code in this repository.
 
-This application uses the pre-installed ADRIA package, and can be built on *either* of `adria-base` or `adria-dev`,
+This application uses the pre-installed ADRIA package, and can be built on _either_ of `adria-base` or `adria-dev`,
 depending on whether you want the development version of the package or not.
 
 It depends on input and output data files which must be provided at runtime.
@@ -280,7 +287,6 @@ using [bind mounts](https://docs.docker.com/storage/bind-mounts/) to data-direct
 your current working directory for this purpose, but [docker volumes](https://docs.docker.com/storage/volumes/)
 with any supported storage driver should also work fine.
 
-
 ```bash
 # EITHER using docker compose (which has the bind-mounts predefined):
 docker compose up adria-sandbox
@@ -327,4 +333,4 @@ julia --project=@. dev.jl
 ```
 
 **Warning:** For `julia` commands, you will probably need to use the `--project=@.` argument. This tells Julia that it's working environment is based in a parent directory of the one the sandbox source code is installed to, which is where the build
-configured all the precompiled dependencies.   If you omit this, the pre-installed packages may not all be available.
+configured all the precompiled dependencies. If you omit this, the pre-installed packages may not all be available.
diff --git a/docker-compose.yaml b/docker-compose.yaml
@@ -38,6 +38,7 @@ services:
       - ./output:/data/output
       - ./sandbox/:/opt/adria-sandbox/src/
       - ./src:/usr/local/src/adria/src
+      - ./assets:/usr/local/src/adria/assets
       - ./test:/usr/local/src/adria/test
       - ./Aviz:/usr/local/src/adria/Aviz
       - ./ext:/usr/local/src/adria/ext