Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates to container images #530

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions .github/workflows/container-images.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
name: container-images

on:
push:
branches:
- main
tags:
- '**'
pull_request:
branches:
- main
Comment on lines +9 to +11
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this will work since you are trying to use the GITHUB_TOKEN secret in an environment which is controlled by the pull request author, isn't it? This would need to be pull_request_target.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GITHUB_TOKEN is only used in steps with the condition github.event_name == 'push'. The idea of including pull_request is to check if the container image still builds with the PR.


jobs:
build-and-push:
runs-on: ubuntu-latest

env:
DOCKER_BUILDX_PLATFORM: linux/amd64

steps:
- uses: actions/checkout@v4

- name: set up Docker buildx
uses: docker/setup-buildx-action@v3
with:
platforms: ${{ env.DOCKER_BUILDX_PLATFORM }}

- uses: docker/login-action@v3
if: ${{ github.event_name == 'push' }}
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: info
run: |
docker version
docker info

echo '${{ github.ref_name }}' | sed -e 's/[^a-zA-Z0-9._-]/_/g' > VERSION_TAG
echo "version_tag=$(cat VERSION_TAG)"

- name: build
run: |
docker buildx build \
--load \
--platform "${DOCKER_BUILDX_PLATFORM}" \
\
--build-arg "VERSION=${{ github.ref_name }}" \
--build-arg "FFMPEG_VERSION=release" \
--build-arg "BUILD_DATE=$(date -u +"%Y-%m-%dT%TZ")" \
--build-arg "GIT_COMMIT=${{ github.sha }}" \
\
-t "ghcr.io/opencast/pyca:latest" \
-t "ghcr.io/opencast/pyca:main" \
Comment on lines +54 to +55
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will create latest and main tagged container images from pull request branches. They aren't pushed, but this might still be confusing.

I think that's what these docker Actions take care of:
https://github.com/opencast/opencast-admin-interface/blob/aec24429505cdd9d12f4587b027ed916a7090c11/.github/workflows/deploy-container-image.yaml#L32-L44

But to be fair, I just copied them from a college who ensured me that this is what I wanted :D

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you said, this only tags images within the build environment. In my CI pipelines, I usually tag images with any potential tag and push if necessary. For this reason, I don't use docker/build-push-action directly, as I want to control if and what tags are pushed.

-t "ghcr.io/opencast/pyca:${{ github.sha }}" \
-t "ghcr.io/opencast/pyca:$(cat VERSION_TAG)" \
.

- name: push release
if: ${{ github.event_name == 'push' && github.ref_type == 'tag' }}
run: |
docker push "ghcr.io/opencast/pyca:$(cat VERSION_TAG)"
# assumption: last tag is always latest version
docker push "ghcr.io/opencast/pyca:latest"

- name: push dev version
if: ${{ github.event_name == 'push' && github.ref_name == 'main' }}
run: |
docker push "ghcr.io/opencast/pyca:main"

- name: delete untagged container images
uses: snok/[email protected]
if: ${{ github.event_name == 'push' }}
with:
account: opencast
token: ${{ secrets.GITHUB_TOKEN }}
tag-selection: untagged
image-names: pyca
cut-off: 1y
103 changes: 63 additions & 40 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
FROM alpine:3.13 AS build

RUN apk --no-cache add \
FROM docker.io/library/alpine:3.20 AS base
RUN apk --no-cache --virtual .run-deps add \
libcurl \
postgresql-libs \
py3-pip \
python3
RUN apk --no-cache --virtual .build-deps add \
curl-dev \
g++ \
gcc \
Expand All @@ -9,63 +13,82 @@ RUN apk --no-cache add \
musl-dev \
nodejs \
npm \
py3-pip \
python3 \
python3-dev \
util-linux \
&& ln -s /usr/bin/python3 /usr/bin/python
postgresql-dev \
python3-dev
RUN pip install --break-system-packages \
gunicorn \
psycopg2

WORKDIR /usr/local/src

FROM base as build-pyca
WORKDIR /usr/local/src
COPY requirements.txt package.json package-lock.json ./
RUN pip install -r requirements.txt \
&& npm i

RUN pip install --break-system-packages -r requirements.txt
RUN npm ci
COPY . .
RUN make pypi

FROM alpine:3.13
LABEL maintainer="pyCA team"

COPY --from=build /usr/local/src/dist/pyca-*.tar.gz /tmp/pyca.tar.gz
FROM base as build

RUN apk --no-cache --virtual .run-deps add \
libcurl \
postgresql-libs \
py3-pip \
python3 \
&& apk --no-cache --virtual .build-deps add \
COPY --from=build-pyca /usr/local/src/dist/pyca-*.tar.gz /tmp/pyca.tar.gz
RUN pip install --break-system-packages \
/tmp/pyca.tar.gz
RUN apk del .build-deps \
&& rm /tmp/pyca.tar.gz


FROM docker.io/library/alpine:3.20 AS build-ffmpeg
ARG TARGETARCH
ARG FFMPEG_VERSION=release
RUN apk add --no-cache \
curl \
curl-dev \
g++ \
gcc \
linux-headers \
make \
musl-dev \
postgresql-dev \
python3-dev \
tar \
xz \
&& ln -s /usr/bin/python3 /usr/bin/python \
&& pip install \
/tmp/pyca.tar.gz \
gunicorn \
psycopg2 \
&& cd /usr/local/bin \
&& curl -sSL "https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz" \
&& mkdir -p /tmp/ffmpeg \
&& cd /tmp/ffmpeg \
&& curl -sSL "https://s3.opencast.org/opencast-ffmpeg-static/ffmpeg-${FFMPEG_VERSION}-${TARGETARCH}-static.tar.xz" \
| tar xJf - --strip-components 1 --wildcards '*/ffmpeg' '*/ffprobe' \
&& apk del .build-deps \
&& rm -rf /tmp/pyca.tar.gz
&& chown root:root ff* \
&& mv ff* /usr/local/bin


FROM scratch AS assembly
COPY --from=build / /
COPY --from=build-ffmpeg /usr/local/bin/ff* /usr/local/bin/

COPY etc/pyca.conf etc/gunicorn.conf.py /etc/pyca/
RUN echo 'bind = "0.0.0.0:8000"' >> /etc/pyca/gunicorn.conf.py

RUN addgroup -S -g 800 pyca \
&& adduser -S -D -h /var/lib/pyca -G pyca -u 800 pyca \
&& addgroup pyca audio \
&& addgroup pyca video

COPY etc/pyca.conf etc/gunicorn.conf.py /etc/pyca/
RUN echo 'bind = "0.0.0.0:8000"' >> /etc/pyca/gunicorn.conf.py

FROM scratch AS squash
LABEL org.opencontainers.image.base.name="docker.io/library/alpine:3.20"

COPY --from=assembly / /
WORKDIR /var/lib/pyca

ARG VERSION=main
ARG BUILD_DATE=unknown
ARG GIT_COMMIT=unknown

LABEL maintainer="pyCA team" \
org.opencontainers.image.title="pyCA" \
org.opencontainers.image.description="Python Capture Agent for Opencast" \
org.opencontainers.image.version="${VERSION}" \
org.opencontainers.image.vendor="Opencast" \
org.opencontainers.image.authors="pyCA team" \
org.opencontainers.image.licenses="LGPL-3.0-only" \
org.opencontainers.image.url="https://github.com/opencast/pyCA/blob/${VERSION}/README.rst" \
org.opencontainers.image.documentation="https://github.com/opencast/pyCA/blob/${VERSION}/README.rst" \
org.opencontainers.image.source="https://github.com/opencast/pyCA" \
org.opencontainers.image.created="${BUILD_DATE}" \
org.opencontainers.image.revision="${GIT_COMMIT}"

USER pyca
VOLUME [ "/var/lib/pyca" ]
EXPOSE 8000
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ pypi: clean build
@printf "\nUpload to PyPI with \"twine upload dist/$$(python setup.py --fullname).tar.gz\"\n"

docker:
@docker build -t quay.io/opencast/pyca .
@docker build -t ghcr.io/opencast/pyca .

clean:
@python setup.py clean --all
Expand Down
2 changes: 1 addition & 1 deletion docs/install/container.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ Install PyCA via Container

PyCA containers will automatically be built for each release and commit.
This can be used to easily deploy pyCA e.g. for capturing network streams.
The containers can be found at `quay.io/repository/opencast/pyca <https://quay.io/repository/opencast/pyca>`_.
The containers can be found at `ghcr.io/opencast/pyca <https://github.com/opencast/pyca/pkgs/container/pyca>`_.


Compose Files
Expand Down
4 changes: 2 additions & 2 deletions init/container/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ PyCA + SQLite

cp etc/pyca.conf init/container/pyca.conf
sed -i "s|#name .*|name = pyca-container|g" init/container/pyca.conf
docker-compose -f init/container/docker-compose.sqlite.yml up
docker compose -f init/container/docker-compose.sqlite.yml up

PyCA + PostgreSQL
-------------
Expand All @@ -20,4 +20,4 @@ PyCA + PostgreSQL
cp etc/pyca.conf init/container/pyca.conf
sed -i "s|#name .*|name = pyca-container|g" init/container/pyca.conf
sed -i "s|#database .*|database = postgresql://pyca:pyca@database/pyca|g" init/container/pyca.conf
docker-compose -f init/container/docker-compose.postgres.yml up
docker compose -f init/container/docker-compose.postgres.yml up
12 changes: 6 additions & 6 deletions init/container/docker-compose.postgres.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,39 +7,39 @@ volumes:
services:
pyca-schedule:
command: schedule
image: quay.io/opencast/pyca
image: ghcr.io/opencast/pyca
restart: always
volumes:
- ./pyca.conf:/etc/pyca/pyca.conf:ro
- pyca:/var/lib/pyca

pyca-ingest:
command: ingest
image: quay.io/opencast/pyca
image: ghcr.io/opencast/pyca
restart: always
volumes:
- ./pyca.conf:/etc/pyca/pyca.conf:ro
- pyca:/var/lib/pyca

pyca-capture:
command: capture
image: quay.io/opencast/pyca
image: ghcr.io/opencast/pyca
restart: always
volumes:
- ./pyca.conf:/etc/pyca/pyca.conf:ro
- pyca:/var/lib/pyca

pyca-agentstate:
command: agentstate
image: quay.io/opencast/pyca
image: ghcr.io/opencast/pyca
restart: always
volumes:
- ./pyca.conf:/etc/pyca/pyca.conf:ro
- pyca:/var/lib/pyca

pyca-ui:
entrypoint: ["gunicorn", "--config=/etc/pyca/gunicorn.conf.py", "pyca.ui:app"]
image: quay.io/opencast/pyca
image: ghcr.io/opencast/pyca
restart: always
volumes:
- ./pyca.conf:/etc/pyca/pyca.conf:ro
Expand All @@ -48,7 +48,7 @@ services:
- "8000:8000"

database:
image: postgres:12.3
image: docker.io/library/postgres:latest
restart: always
environment:
- POSTGRES_PASSWORD=pyca
Expand Down
10 changes: 5 additions & 5 deletions init/container/docker-compose.sqlite.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,39 +6,39 @@ volumes:
services:
pyca-schedule:
command: schedule
image: quay.io/opencast/pyca
image: ghcr.io/opencast/pyca
restart: always
volumes:
- ./pyca.conf:/etc/pyca/pyca.conf:ro
- pyca:/var/lib/pyca

pyca-ingest:
command: ingest
image: quay.io/opencast/pyca
image: ghcr.io/opencast/pyca
restart: always
volumes:
- ./pyca.conf:/etc/pyca/pyca.conf:ro
- pyca:/var/lib/pyca

pyca-capture:
command: capture
image: quay.io/opencast/pyca
image: ghcr.io/opencast/pyca
restart: always
volumes:
- ./pyca.conf:/etc/pyca/pyca.conf:ro
- pyca:/var/lib/pyca

pyca-agentstate:
command: agentstate
image: quay.io/opencast/pyca
image: ghcr.io/opencast/pyca
restart: always
volumes:
- ./pyca.conf:/etc/pyca/pyca.conf:ro
- pyca:/var/lib/pyca

pyca-ui:
entrypoint: ["gunicorn", "--config=/etc/pyca/gunicorn.conf.py", "pyca.ui:app"]
image: quay.io/opencast/pyca
image: ghcr.io/opencast/pyca
restart: always
volumes:
- ./pyca.conf:/etc/pyca/pyca.conf:ro
Expand Down
Loading