breeze ci resource-check hangs with a python prompt

[tomicooler]

Issue description

I have an issue with the breeze dev tool. When I run the breeze ci resource-check command the process just hangs with a python prompt.

I tried to debug, with verbose_override=True:

def check_docker_resources(airflow_image_name: str) -> RunCommandResult:
    """
    Check if we have enough resources to run docker. This is done via running script embedded in our image.


    :param airflow_image_name: name of the airflow image to use
    """
    return run_command(
        cmd=[
            "docker",
            "run",
            "-t",
            "--entrypoint",
            "/bin/bash",
            "-e",
            "PYTHONDONTWRITEBYTECODE=true",
            airflow_image_name,
            "-c",
            "python /opt/airflow/scripts/in_container/run_resource_check.py",
        ],
        text=True,
        verbose_override=True,
    )

The output:

+ docker version
Client:
 Version:           27.2.1
 API version:       1.46 (downgraded from 1.47)
 Go version:        go1.23.1
 Git commit:        9e34c9b
 Built:             Tue Sep 10 23:00:53 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Desktop 4.33.0 (160616)
 Engine:
  Version:          27.1.1
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.12
  Git commit:       cc13f95
  Built:            Tue Jul 23 19:57:14 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.19
  GitCommit:        2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc:
  Version:          1.7.19
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
+ breeze ci resource-check
Good version of Docker: 27.2.1.
Good version of docker-compose: 2.29.3
Executable permissions on entrypoints are OK

Running command: <docker run ghcr.io/apache/airflow/v2-9-test/ci/python3.8 'python /opt/airflow/scripts/in_container/run_resource_check.py'>


Working directory /airflow


docker run -t --entrypoint /bin/bash -e PYTHONDONTWRITEBYTECODE=true ghcr.io/apache/airflow/v2-9-test/ci/python3.8 -c 'python /opt/airflow/scripts/in_container/run_resource_check.py'

Python 3.8.19 (default, Jul 23 2024, 07:18:03)
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

$ docker inspect 068715446c2d
[
    {
        "Id": "068715446c2d701c31e449c971f0abfa2e8a380a5da2fb26b3ab75c3f6766c3b",
        "Created": "2024-09-16T09:25:46.295298013Z",
        "Path": "/bin/bash",
        "Args": [
            "-c",
            "python",
            "/opt/airflow/scripts/in_container/run_resource_check.py"
        ],

The python /opt/airflow/scripts/in_container/run_resource_check.py should be one parameter. Even though breeze logs that the parameter after -c is quoted, it is probably not.

Reproduction and debug

I can repro a similar issue when running locally like this:

$ docker run -ti --platform=linux/amd64 --entrypoint "/bin/bash" -e PYTHONDONTWRITEBYTECODE=true  ghcr.io/apache/airflow/v2-9-test/ci/python3.8:latest -c python /opt/airflow/scripts/in_container/run_resource_check.py
Python 3.8.19 (default, Jul 23 2024, 07:18:03)
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

$ docker inspect bd7f0bf81a5b
[
    {
        "Id": "bd7f0bf81a5b1f4763e667d9056e8c7802df0378c028cdb1d35388a9388486d2",
        "Created": "2024-09-16T11:03:11.620829969Z",
        "Path": "/bin/bash",
        "Args": [
            "-c",
            "python",
            "/opt/airflow/scripts/in_container/run_resource_check.py"
        ],

$ docker run -ti --platform=linux/amd64 --entrypoint "/bin/bash" -e PYTHONDONTWRITEBYTECODE=true  ghcr.io/apache/airflow/v2-9-test/ci/python3.8:latest -c 'python /opt/airflow/scripts/in_container/run_resource_check.py'

Checking resources.

 * Memory available 17GB. OK.
 * Cpus available 12. OK.
WARNING!!!: Not enough Disk available for Docker.
At least 20GB of Disk required.  You have 0GB

WARNING!!!: You have not enough resources to run Airflow (see above)!
Please follow the instructions to increase amount of resources available:
 Please check https://github.com/apache/airflow/blob/main/dev/breeze/doc/01_installation.rst#resources-required for details

The only difference between the two command run is that the parameter for bash -c is in ''.

Example with a sleep command, to see the args in docker inspect:

$ docker run -ti --platform=linux/amd64 --entrypoint "/bin/bash" -e PYTHONDONTWRITEBYTECODE=true  ghcr.io/apache/airflow/v2-9-test/ci/python3.8:latest -c 'sleep 1000'

$ docker inspect fcb06f9293c4
[
    {
        "Id": "fcb06f9293c46154152e61f47eb895253f995dd09170bfcabe25a5b2a0234f9d",
        "Created": "2024-09-16T11:05:10.509913843Z",
        "Path": "/bin/bash",
        "Args": [
            "-c",
            "sleep 1000"
        ],

Docker version:

Client: Docker Engine - Community
 Version:           27.1.2
 API version:       1.46
 Go version:        go1.22.6
 Git commit:        d01f264bcc
 Built:             Mon Aug 12 11:34:32 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.33.0 (160616)
 Engine:
  Version:          27.1.1
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.12
  Git commit:       cc13f95
  Built:            Tue Jul 23 19:57:14 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.19
  GitCommit:        2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc:
  Version:          1.7.19
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

The command ends up in a subprocess.run:

airflow/dev/breeze/src/airflow_breeze/utils/run_utils.py

Line 161 in a5d0a63

return subprocess.run(cmd, input=input, check=check, env=cmd_env, cwd=workdir, **kwargs)

Workaround

As a workaround I changed the entrypoint to /usr/local/bin/python:

def check_docker_resources(airflow_image_name: str) -> RunCommandResult:
    """
    Check if we have enough resources to run docker. This is done via running script embedded in our image.


    :param airflow_image_name: name of the airflow image to use
    """
    return run_command(
        cmd=[
            "docker",
            "run",
            "-t",
            "--entrypoint",
            "/usr/local/bin/python",
            "-e",
            "PYTHONDONTWRITEBYTECODE=true",
            airflow_image_name,
            "/opt/airflow/scripts/in_container/run_resource_check.py",
        ],
        text=True,
    )

[potiuk]

Likely your problem is lack of resources (disk space). You can clearly see that

 * Memory available 17GB. OK.
 * Cpus available 12. OK.
WARNING!!!: Not enough Disk available for Docker.
At least 20GB of Disk required.  You have 0GB

Your Docker does not have free disk you must free some space.