Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Alpine to python feature #1015

Open
wants to merge 55 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 44 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
77d36ba
fix: don't assume `yum` is available by default; otherwise, fail script
jonbackhaus Jun 22, 2024
0d22f3c
feat: baseline alpine config, based on existing debian config and APK…
jonbackhaus Jun 22, 2024
da8f5f7
fix: remove nested quotes
jonbackhaus Jun 22, 2024
01af53f
fix: add baseline packages
jonbackhaus Jun 22, 2024
8659cc6
feat: support Alpine user/group management commands
jonbackhaus Jun 22, 2024
e121aa4
fix: wrong path for apk cache
jonbackhaus Jun 22, 2024
ad27565
fix: wrong command for adding a user to a group in Alpine
jonbackhaus Jun 22, 2024
4c3fe15
fix: these packages don't exist in Alpine
jonbackhaus Jun 23, 2024
2b94a42
fix: use `ack` for Alpine, since `grep` doesn't support PCRE
jonbackhaus Jun 23, 2024
37d0cb9
fix: add `gpg`, `gpg-agent` packages (required for package signature …
jonbackhaus Jun 23, 2024
0d5405b
perf: swap `alpine-sdk` for individual packages, from alpine docker i…
jonbackhaus Jun 23, 2024
d313b08
refactor: switch `case` statement to `if`
jonbackhaus Jun 23, 2024
5cb60e6
feat: add build from source support for Alpine
jonbackhaus Jun 23, 2024
125d71b
refactor: only install gpg, gpg-agent packages if building from source
jonbackhaus Jun 23, 2024
f3740fa
feat: fallback to non-relative symlink if not supported (e.g., Alpine…
jonbackhaus Jun 23, 2024
6ef78c7
chore: bump minor revision (proposed)
jonbackhaus Jun 23, 2024
170f53c
docs: add Alpine/apk to supported distros
jonbackhaus Jun 23, 2024
c58fa35
test: add test case for additional JupyterLab (Alpine)
jonbackhaus Jun 23, 2024
b283c0d
test: add test case for additional Python version/s (Alpine)
jonbackhaus Jun 23, 2024
29b7c22
test: add test case for alternate Python tools (Alpine)
jonbackhaus Jun 23, 2024
4159f5f
test: add test case for JupyterLab (Alpine)
jonbackhaus Jun 23, 2024
19598ea
test: add test case for OS-provided Python (Alpine)
jonbackhaus Jun 23, 2024
b041608
test: add test case for Python shared lib (Alpine)
jonbackhaus Jun 23, 2024
10a47e1
fix: use Devcontainer Alpine image as base (because it has `bash` ins…
jonbackhaus Jun 24, 2024
2f6af5f
revert: switch back to PCRE-based grep for consistency
jonbackhaus Jun 27, 2024
4715c20
refactor: move definition closer to usage
jonbackhaus Jun 27, 2024
99e118b
refactor: define formal functions vs. variables as pseudo-functions
jonbackhaus Jun 27, 2024
1866bd9
chore: remove OBE `ack` package (missed in earlier commit)
jonbackhaus Jun 27, 2024
ab54f4d
Merge branch 'main' into python-alpine
jonbackhaus Jun 27, 2024
866d03e
Merge branch 'python-alpine' of github.com:jonbackhaus/devcontainer-f…
jonbackhaus Jun 27, 2024
cd1d7ad
fix: remove `--with-lto` option
jonbackhaus Jun 27, 2024
f6be63d
fix: remove `--enable-loadable-sqlite-extensions` option
jonbackhaus Jun 27, 2024
da59638
fix: remove `--with-system-expat` option
jonbackhaus Jun 27, 2024
71c0b31
fix: remove `--build` option
jonbackhaus Jun 27, 2024
e992a23
fix: remove `-DTHREAD_STACK_SIZE` EXTRA_CFLAGS option
jonbackhaus Jun 27, 2024
cecf5f4
fix: remove `--enable-option-checking=fatal` option
jonbackhaus Jun 27, 2024
f05ad20
revert: remove flags (now OBE)
jonbackhaus Jun 27, 2024
c7609fd
chore: remove OBE packages
jonbackhaus Jun 27, 2024
7393df3
style: reorder tests and clarify pip vs pip3
jonbackhaus Jun 28, 2024
9838678
fix: add Alpine-specific packages for system/os-provided package
jonbackhaus Jun 28, 2024
8ef3f47
test: fix: switch to non-root user for Alpine (similar to RHEL)
jonbackhaus Jun 28, 2024
ab58446
test: fix: switch to generic Alpine base
jonbackhaus Jun 28, 2024
4adeb89
test: fix: switch to generic Alpine base
jonbackhaus Jun 28, 2024
94b0bf1
Merge branch 'main' into python-alpine
jonbackhaus Aug 10, 2024
565deff
Update test/python/scenarios.json
jonbackhaus Aug 18, 2024
4ab847a
Merge branch 'main' into python-alpine
jonbackhaus Aug 18, 2024
4a81865
test: fix: resolve inconsistencies in test configuration
jonbackhaus Aug 18, 2024
c2b546f
fix: force clean before initial package check/install, just in case t…
jonbackhaus Aug 18, 2024
64713e1
fix: improve handling of grep install
jonbackhaus Aug 21, 2024
d25bf76
Merge branch 'main' into python-alpine
jonbackhaus Aug 21, 2024
395e768
Merge branch 'main' into python-alpine
jonbackhaus Aug 26, 2024
ec8ec36
test: temporarily removing 'centos-7' test (fixed in another PR)
jonbackhaus Aug 26, 2024
acc390b
Merge branch 'main' into python-alpine
jonbackhaus Aug 28, 2024
2ea3352
Merge branch 'main' into python-alpine
jonbackhaus Sep 7, 2024
b113acb
Merge branch 'main' into python-alpine
jonbackhaus Sep 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/python/NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@

jonbackhaus marked this conversation as resolved.
Show resolved Hide resolved
## OS Support

This Feature should work on recent versions of Debian/Ubuntu, RedHat Enterprise Linux, Fedora, Alma, and RockyLinux distributions with the apt, yum, dnf, or microdnf package manager installed.
This Feature should work on recent versions of Alpine, Debian/Ubuntu, RedHat Enterprise Linux, Fedora, Alma, and RockyLinux distributions with the apk, apt, yum, dnf, or microdnf package manager installed.

`bash` is required to execute the `install.sh` script.
8 changes: 4 additions & 4 deletions src/python/devcontainer-feature.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"id": "python",
"version": "1.6.3",
"version": "1.7.0",
"name": "Python",
"documentationURL": "https://github.com/devcontainers/features/tree/main/src/python",
"description": "Installs the provided version of Python, as well as PIPX, and other common Python utilities. JupyterLab is conditionally installed with the python feature. Note: May require source code compilation.",
Expand Down Expand Up @@ -78,9 +78,9 @@
],
"settings": {
"python.defaultInterpreterPath": "/usr/local/python/current/bin/python",
"[python]": {
"editor.defaultFormatter": "ms-python.autopep8"
}
"[python]": {
"editor.defaultFormatter": "ms-python.autopep8"
}
}
}
},
Expand Down
113 changes: 99 additions & 14 deletions src/python/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@ fi
MAJOR_VERSION_ID=$(echo ${VERSION_ID} | cut -d . -f 1)
if [ "${ID}" = "debian" ] || [ "${ID_LIKE}" = "debian" ]; then
ADJUSTED_ID="debian"
elif [ "${ID}" = "alpine" ] || [ "${ID_LIKE}" = "alpine" ]; then
jonbackhaus marked this conversation as resolved.
Show resolved Hide resolved
ADJUSTED_ID="alpine"
elif [[ "${ID}" = "rhel" || "${ID}" = "fedora" || "${ID}" = "mariner" || "${ID_LIKE}" = *"rhel"* || "${ID_LIKE}" = *"fedora"* || "${ID_LIKE}" = *"mariner"* ]]; then
ADJUSTED_ID="rhel"
if [[ "${ID}" = "rhel" ]] || [[ "${ID}" = *"alma"* ]] || [[ "${ID}" = *"rocky"* ]]; then
Expand All @@ -74,20 +76,29 @@ fi
if type apt-get > /dev/null 2>&1; then
PKG_MGR_CMD=apt-get
INSTALL_CMD="${PKG_MGR_CMD} -y install --no-install-recommends"
elif type apk > /dev/null 2>&1; then
PKG_MGR_CMD=apk
INSTALL_CMD="${PKG_MGR_CMD} add --no-cache --no-interactive"
elif type microdnf > /dev/null 2>&1; then
PKG_MGR_CMD=microdnf
INSTALL_CMD="${PKG_MGR_CMD} ${INSTALL_CMD_ADDL_REPOS} -y install --refresh --best --nodocs --noplugins --setopt=install_weak_deps=0"
elif type dnf > /dev/null 2>&1; then
PKG_MGR_CMD=dnf
INSTALL_CMD="${PKG_MGR_CMD} ${INSTALL_CMD_ADDL_REPOS} -y install --refresh --best --nodocs --noplugins --setopt=install_weak_deps=0"
else
elif type yum > /dev/null 2>&1; then
PKG_MGR_CMD=yum
INSTALL_CMD="${PKG_MGR_CMD} ${INSTALL_CMD_ADDL_REPOS} -y install --noplugins --setopt=install_weak_deps=0"
else
echo "(Error) Unable to find a supported package manager."
exit 1
fi

# Clean up
clean_up() {
case ${ADJUSTED_ID} in
alpine)
rm -rf /var/cache/apk/*
;;
debian)
rm -rf /var/lib/apt/lists/*
;;
Expand All @@ -107,6 +118,10 @@ updaterc() {
local _zshrc
if [ "${UPDATE_RC}" = "true" ]; then
case $ADJUSTED_ID in
alpine) echo "Updating /etc/bash/bashrc and /etc/zsh/zshrc..."
_bashrc=/etc/bash/bashrc
_zshrc=/etc/zsh/zshrc
;;
debian) echo "Updating /etc/bash.bashrc and /etc/zsh/zshrc..."
_bashrc=/etc/bash.bashrc
_zshrc=/etc/zsh/zshrc
Expand Down Expand Up @@ -367,6 +382,12 @@ oryx_install() {

pkg_mgr_update() {
case $ADJUSTED_ID in
alpine)
if [ "$(find /var/cache/apk/* | wc -l)" = "0" ]; then
echo "Running apk update..."
${PKG_MGR_CMD} update
fi
;;
debian)
if [ "$(find /var/lib/apt/lists/* | wc -l)" = "0" ]; then
echo "Running apt-get update..."
Expand Down Expand Up @@ -398,6 +419,12 @@ pkg_mgr_update() {
# Checks if packages are installed and installs them if not
check_packages() {
case ${ADJUSTED_ID} in
alpine)
if ! apk info --installed "$@" > /dev/null 2>&1; then
pkg_mgr_update
${INSTALL_CMD} "$@"
fi
;;
debian)
if ! dpkg -s "$@" > /dev/null 2>&1; then
pkg_mgr_update
Expand All @@ -415,13 +442,13 @@ check_packages() {

add_symlink() {
if [[ ! -d "${CURRENT_PATH}" ]]; then
ln -s -r "${INSTALL_PATH}" "${CURRENT_PATH}"
ln -s -r "${INSTALL_PATH}" "${CURRENT_PATH}" || ln -s "${INSTALL_PATH}" "${CURRENT_PATH}"
fi

if [ "${OVERRIDE_DEFAULT_VERSION}" = "true" ]; then
if [[ $(ls -l ${CURRENT_PATH}) != *"-> ${INSTALL_PATH}"* ]] ; then
rm "${CURRENT_PATH}"
ln -s -r "${INSTALL_PATH}" "${CURRENT_PATH}"
ln -s -r "${INSTALL_PATH}" "${CURRENT_PATH}" || ln -s "${INSTALL_PATH}" "${CURRENT_PATH}"
fi
fi
}
Expand Down Expand Up @@ -483,13 +510,13 @@ install_from_source() {
# via common package repositories, for now rhel-7 family, use case statement to
# make it easy to expand
SSL_INSTALL_PATH="/usr/local"
case ${VERSION_CODENAME} in
centos7|rhel7)
check_packages perl-IPC-Cmd
install_openssl3
ADDL_CONFIG_ARGS="--with-openssl=${SSL_INSTALL_PATH} --with-openssl-rpath=${SSL_INSTALL_PATH}/lib"
;;
esac
if [ "${VERSION_CODENAME}" = "centos7" ] || [ "${VERSION_CODENAME}" = "rhel7" ]; then
check_packages perl-IPC-Cmd
install_openssl3
ADDL_CONFIG_ARGS="--with-openssl=${SSL_INSTALL_PATH} --with-openssl-rpath=${SSL_INSTALL_PATH}/lib"
elif [ "${ADJUSTED_ID}" = "alpine" ]; then
check_packages gpg gpg-agent
fi

install_cpython "${VERSION}"
if [ -f "/tmp/python-src/${cpython_tgz_filename}" ]; then
Expand Down Expand Up @@ -606,6 +633,8 @@ install_python() {
if [ ${version} = "os-provided" ] || [ ${version} = "system" ]; then
if [ ${ADJUSTED_ID} = "debian" ]; then
check_packages python3 python3-doc python3-pip python3-venv python3-dev python3-tk
elif [ ${ADJUSTED_ID} = "alpine" ]; then
check_packages python3 python3-doc py3-pip py3-virtualenv python3-dev python3-tkinter
else
if [ ${ID} != "mariner" ]; then
check_packages python3 python3-pip python3-devel python3-tkinter
Expand Down Expand Up @@ -656,6 +685,27 @@ sys.prefix == sys.base_prefix and print(sysconfig.get_path("stdlib", sysconfig.g
fi
}

add_system_group() {
local _group=$1

if [ "${ADJUSTED_ID}" = "alpine" ]; then
addgroup --system "${_group}"
else
groupadd --system "${_group}"
fi
}

add_user_to_group() {
local _user=$1
local _group=$2

if [ "${ADJUSTED_ID}" = "alpine" ]; then
addgroup "${_user}" "${_group}"
else
usermod --append --groups "${_group}" "${_user}"
fi
}

# Ensure that login shells get the correct path if the user updated the PATH using ENV.
rm -f /etc/profile.d/00-restore-env.sh
echo "export PATH=${PATH//$(sh -lc 'echo $PATH')/\$PATH}" > /etc/profile.d/00-restore-env.sh
Expand All @@ -666,6 +716,9 @@ if ! type awk >/dev/null 2>&1; then
check_packages awk
fi

# Some distributions do not include a PCRE-enabled grep by default (e.g., Alpine)
${INSTALL_CMD} --upgrade grep

# Determine the appropriate non-root user
if [ "${USERNAME}" = "auto" ] || [ "${USERNAME}" = "automatic" ]; then
USERNAME=""
Expand All @@ -690,6 +743,38 @@ export DEBIAN_FRONTEND=noninteractive

REQUIRED_PKGS=""
case ${ADJUSTED_ID} in
alpine)
REQUIRED_PKGS="${REQUIRED_PKGS} \
curl \
"

# ref. <https://github.com/docker-library/python/blob/2d4fb586c48b067b432cf56653ee2541d94fdd7d/3.11/alpine3.20/Dockerfile#L29>
REQUIRED_PKGS="${REQUIRED_PKGS} \
bluez-dev \
bzip2-dev \
expat-dev \
findutils \
gcc \
gdbm-dev \
libc-dev \
libffi-dev \
libnsl-dev \
libtirpc-dev \
linux-headers \
make \
ncurses-dev \
openssl-dev \
pax-utils \
readline-dev \
sqlite-dev \
tcl-dev \
tk \
tk-dev \
util-linux-dev \
xz-dev \
zlib-dev \
"
;;
debian)
REQUIRED_PKGS="${REQUIRED_PKGS} \
ca-certificates \
Expand Down Expand Up @@ -759,9 +844,9 @@ check_packages ${REQUIRED_PKGS}
# Install Python from source if needed
if [ "${PYTHON_VERSION}" != "none" ]; then
if ! cat /etc/group | grep -e "^python:" > /dev/null 2>&1; then
groupadd -r python
add_system_group python
fi
usermod -a -G python "${USERNAME}"
add_user_to_group "${USERNAME}" python

CURRENT_PATH="${PYTHON_INSTALL_PATH}/current"

Expand Down Expand Up @@ -807,9 +892,9 @@ if [[ "${INSTALL_PYTHON_TOOLS}" = "true" ]] && [[ -n "${PYTHON_SRC}" ]]; then

# Create pipx group, dir, and set sticky bit
if ! cat /etc/group | grep -e "^pipx:" > /dev/null 2>&1; then
groupadd -r pipx
add_system_group pipx
fi
usermod -a -G pipx ${USERNAME}
add_user_to_group "${USERNAME}" pipx
umask 0002
mkdir -p ${PIPX_BIN_DIR}
chown -R "${USERNAME}:pipx" ${PIPX_HOME}
Expand Down
40 changes: 40 additions & 0 deletions test/python/alpine.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#!/bin/bash

set -e

# Optional: Import test library
source dev-container-features-test-lib

# Definition specific tests
check "version" python --version
check "pip is installed" pip --version
check "pip is installed" pip3 --version

# Check that tools can execute
check "autopep8" autopep8 --version
check "black" black --version
check "yapf" yapf --version
check "bandit" bandit --version
check "flake8" flake8 --version
check "mypy" mypy --version
check "pycodestyle" pycodestyle --version
check "pydocstyle" pydocstyle --version
check "pylint" pylint --version
check "pytest" pytest --version

# Check paths in settings
check "current symlink is correct" bash -c "which python | grep /usr/local/python/current/bin/python"
check "current symlink works" /usr/local/python/current/bin/python --version
check "which autopep8" bash -c "which autopep8 | grep /usr/local/py-utils/bin/autopep8"
check "which black" bash -c "which black | grep /usr/local/py-utils/bin/black"
check "which yapf" bash -c "which yapf | grep /usr/local/py-utils/bin/yapf"
check "which bandit" bash -c "which bandit | grep /usr/local/py-utils/bin/bandit"
check "which flake8" bash -c "which flake8 | grep /usr/local/py-utils/bin/flake8"
check "which mypy" bash -c "which mypy | grep /usr/local/py-utils/bin/mypy"
check "which pycodestyle" bash -c "which pycodestyle | grep /usr/local/py-utils/bin/pycodestyle"
check "which pydocstyle" bash -c "which pydocstyle | grep /usr/local/py-utils/bin/pydocstyle"
check "which pylint" bash -c "which pylint | grep /usr/local/py-utils/bin/pylint"
check "which pytest" bash -c "which pytest | grep /usr/local/py-utils/bin/pytest"

# Report result
reportResults
26 changes: 26 additions & 0 deletions test/python/install_additional_jupyterlab_alpine.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/bin/bash

set -e

# Optional: Import test library
source dev-container-features-test-lib

# Always run these checks as the non-root user
user="$(whoami)"
check "user" grep vscode <<< "$user"

# Check for an installation of JupyterLab
check "version" jupyter lab --version

# Check location of JupyterLab installation
packages="$(python3 -m pip list)"
check "location" grep jupyter <<< "$packages"

# Check for git extension
check "jupyterlab_git" grep jupyterlab_git <<< "$packages"

# Check for correct JupyterLab configuration
check "config" grep ".*.allow_origin = '*'" /home/vscode/.jupyter/jupyter_server_config.py

# Report result
reportResults
41 changes: 41 additions & 0 deletions test/python/install_additional_python_alpine.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#!/bin/bash

set -e

# Optional: Import test library
source dev-container-features-test-lib

check "python version 3.11 installed as default" bash -c "python --version | grep 3.11"
check "python3 version 3.11 installed as default" bash -c "python3 --version | grep 3.11"
check "python version 3.10.5 installed" bash -c "ls -l /usr/local/python | grep 3.10.5"
check "python version 3.8 installed" bash -c "ls -l /usr/local/python | grep 3.8"
check "python version 3.9.13 installed" bash -c "ls -l /usr/local/python | grep 3.9.13"

# Check that tools can execute - make sure something didn't get messed up in this scenario
check "autopep8" autopep8 --version
check "black" black --version
check "yapf" yapf --version
check "bandit" bandit --version
check "flake8" flake8 --version
check "mypy" mypy --version
check "pycodestyle" pycodestyle --version
check "pydocstyle" pydocstyle --version
check "pylint" pylint --version
check "pytest" pytest --version

# Check paths in settings
check "current symlink is correct" bash -c "which python | grep /usr/local/python/current/bin/python"
check "current symlink works" /usr/local/python/current/bin/python --version
check "which autopep8" bash -c "which autopep8 | grep /usr/local/py-utils/bin/autopep8"
check "which black" bash -c "which black | grep /usr/local/py-utils/bin/black"
check "which yapf" bash -c "which yapf | grep /usr/local/py-utils/bin/yapf"
check "which bandit" bash -c "which bandit | grep /usr/local/py-utils/bin/bandit"
check "which flake8" bash -c "which flake8 | grep /usr/local/py-utils/bin/flake8"
check "which mypy" bash -c "which mypy | grep /usr/local/py-utils/bin/mypy"
check "which pycodestyle" bash -c "which pycodestyle | grep /usr/local/py-utils/bin/pycodestyle"
check "which pydocstyle" bash -c "which pydocstyle | grep /usr/local/py-utils/bin/pydocstyle"
check "which pylint" bash -c "which pylint | grep /usr/local/py-utils/bin/pylint"
check "which pytest" bash -c "which pytest | grep /usr/local/py-utils/bin/pytest"

# Report result
reportResults
32 changes: 32 additions & 0 deletions test/python/install_alternate_tools_alpine.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
#!/bin/bash

set -e

# Optional: Import test library
source dev-container-features-test-lib

# Definition specific tests
check "version" python --version
check "pip is installed" pip --version
check "pip is installed" pip3 --version

# Check that tools can execute
check "bandit" bandit --version
check "mypy" mypy --version
check "pipenv" pipenv --version
check "pytest" pytest --version
check "ruff" ruff --version
check "virtualenv" virtualenv --version

# Check paths in settings
check "current symlink is correct" bash -c "which python | grep /usr/local/python/current/bin/python"
check "current symlink works" /usr/local/python/current/bin/python --version
check "which bandit" bash -c "which bandit | grep /usr/local/py-utils/bin/bandit"
check "which mypy" bash -c "which mypy | grep /usr/local/py-utils/bin/mypy"
check "which pipenv" bash -c "which pipenv | grep /usr/local/py-utils/bin/pipenv"
check "which pytest" bash -c "which pytest | grep /usr/local/py-utils/bin/pytest"
check "which ruff" bash -c "which ruff | grep /usr/local/py-utils/bin/ruff"
check "which virtualenv" bash -c "which virtualenv | grep /usr/local/py-utils/bin/virtualenv"

# Report result
reportResults
Loading
Loading