Clarifies installation/runtime options for CI/PROD images. (apache#15320)

After PROD images were added, some of the flags had two meanings

These behaved differently in PROD image and CI image and were the
source of confusion especially when start-airflow command was used.

For PROD image, the image can be customized during image building,
and packages could be installed from .whl or .sdist packages
available in `docker-context-files`. This
is used at CI and dockerhub building time to produce image built
packages that were prepared using local sources.

The CI image is always built from local sources but airflow can
be removed and re-installed at runtime from pypi.
Both airflow and provider packages can be installed
from .whl or .sdist packages available in dist folder. This is
used in CI to test current provider packages with older
Airflow released (2.0.0) and to test provider packages locally.

After the change we have two sets of flags/variables:

PROD image (building image):

* install-airflow-version, install-airflow-reference,
  install-from-docker-context-files

CI image (runtime):

* use-airflow-version, use-packages-from-dist

That should avoid confusion and failures of commands such as
`start-airflow` that is used to test provider packages and
airflow itself.

Author: potiuk

.github/workflows/ci.yml

@@ -551,12 +551,12 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
       - name: "Install and test provider packages and airflow via wheel files"
         run: ./scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh
         env:
-          INSTALL_AIRFLOW_VERSION: "wheel"
+          USE_AIRFLOW_VERSION: "wheel"
           PACKAGE_FORMAT: "wheel"
       - name: "Install and test provider packages and airflow on Airflow 2.0 files"
         run: ./scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh
         env:
-          INSTALL_AIRFLOW_VERSION: "2.0.0"
+          USE_AIRFLOW_VERSION: "2.0.0"
           PACKAGE_FORMAT: "wheel"
 
   prepare-test-provider-packages-sdist:
@@ -597,7 +597,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
       - name: "Install and test provider packages and airflow via sdist files"
         run: ./scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh
         env:
-          INSTALL_AIRFLOW_VERSION: "sdist"
+          USE_AIRFLOW_VERSION: "sdist"
           PACKAGE_FORMAT: "sdist"
 
   tests-helm:

BREEZE.rst

@@ -1250,6 +1250,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
         'breeze \
               --github-image-id 209845560' - pull/use image with RUN_ID
 
+  Most flags are applicable to the shell command as it will run build when needed.
+
 
   ####################################################################################################
 
@@ -1310,41 +1312,21 @@ This is the current syntax for  `./breeze <./breeze>`_:
                  2.7 3.5 3.6 3.7 3.8
 
   -a, --install-airflow-version INSTALL_AIRFLOW_VERSION
-          In CI image, installs Airflow (in entrypoint) from PIP released version or using
-          the installation method specified (sdist, wheel, none).
-
-          In PROD image the installation of selected method or version happens during image building.
-          For PROD image, the 'none' options is not valid.
-
-          One of:
+          Uses differen version of Airflow when building PROD image.
 
-                 2.0.1 2.0.0 1.10.15 1.10.14 1.10.12 1.10.11 1.10.10 1.10.9 none wheel sdist
-
-          When 'none' is used, you can install airflow from local packages. When building image,
-          airflow package should be added to 'docker-context-files' and
-          --install-from-docker-context-files flag should be used. When running an image, airflow
-          package should be added to dist folder and --install-packages-from-dist flag should be used.
+                 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist
 
   -t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
-          If specified, installs Airflow directly from reference in GitHub. This happens at
-          image building time in production image and at container entering time for CI image.
+          Installs Airflow directly from reference in GitHub when building PROD image.
           This can be a GitHub branch like master or v1-10-test, or a tag like 2.0.0a1.
 
   --installation-method INSTALLATION_METHOD
-          Method of installing airflow for production image - either from the sources ('.')
+          Method of installing Airflow in PROD image - either from the sources ('.')
           or from package 'apache-airflow' to install from PyPI.
           Default in Breeze is to install from sources. One of:
 
                  . apache-airflow
 
-  --no-rbac-ui
-          Disables RBAC UI when Airflow 1.10.* is installed.
-
-  --install-packages-from-dist
-          If specified it will look for packages placed in dist folder and it will install the
-          packages after installing Airflow. This is useful for testing provider
-          packages.
-
   --upgrade-to-newer-dependencies
           Upgrades PIP packages to latest versions available without looking at the constraints.
 
@@ -1392,12 +1374,12 @@ This is the current syntax for  `./breeze <./breeze>`_:
           are needed by the extras. When you build image during the development (which is
           default in Breeze) all providers are installed by default from sources.
           You can disable it by adding this flag but then you have to install providers from
-          wheel packages via --install-packages-from-dist flag.
+          wheel packages via --use-packages-from-dist flag.
 
   --disable-pypi-when-building
           Disable installing Airflow from pypi when building. If you use this flag and want
           to install Airflow, you have to install it from packages placed in
-          'docker-context-files' and use --install-from-local-files-when-building flag.
+          'docker-context-files' and use --install-from-docker-context-files flag.
 
   --additional-extras ADDITIONAL_EXTRAS
           Additional extras to pass to build images The default is no additional extras.
@@ -1452,7 +1434,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
   --disable-pip-cache
           Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
 
-  --install-from-local-files-when-building
+  --install-from-docker-context-files
           This flag is used during image building. If it is used additionally to installing
           Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
           in the 'docker-context-files' folder. The same flag can be used during entering the image in
@@ -1839,6 +1821,22 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Flags:
 
+  --use-airflow-version AIRFLOW_SPECIFICATION
+          In CI image, installs Airflow at runtime from PIP released version or using
+          the installation method specified (sdist, wheel, none). When 'none' is used,
+          airflow is just removed. In this case airflow package should be added to dist folder
+          and --use-packages-from-dist flag should be used.
+
+                 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist none
+
+  --use-packages-from-dist
+          In CI image, if specified it will look for packages placed in dist folder and
+          it will install the packages after entering the image.
+          This is useful for testing provider packages.
+
+  --no-rbac-ui
+          Disables RBAC UI when Airflow 1.10.* is installed.
+
   --load-example-dags
           Include Airflow example dags.
 
@@ -2043,12 +2041,12 @@ This is the current syntax for  `./breeze <./breeze>`_:
           are needed by the extras. When you build image during the development (which is
           default in Breeze) all providers are installed by default from sources.
           You can disable it by adding this flag but then you have to install providers from
-          wheel packages via --install-packages-from-dist flag.
+          wheel packages via --use-packages-from-dist flag.
 
   --disable-pypi-when-building
           Disable installing Airflow from pypi when building. If you use this flag and want
           to install Airflow, you have to install it from packages placed in
-          'docker-context-files' and use --install-from-local-files-when-building flag.
+          'docker-context-files' and use --install-from-docker-context-files flag.
 
   --additional-extras ADDITIONAL_EXTRAS
           Additional extras to pass to build images The default is no additional extras.
@@ -2103,7 +2101,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
   --disable-pip-cache
           Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
 
-  --install-from-local-files-when-building
+  --install-from-docker-context-files
           This flag is used during image building. If it is used additionally to installing
           Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
           in the 'docker-context-files' folder. The same flag can be used during entering the image in
@@ -2526,50 +2524,49 @@ This is the current syntax for  `./breeze <./breeze>`_:
           Assume 'quit' answer to all questions.
 
   ****************************************************************************************************
-   Choose different Airflow version to install or run
+   Install different Airflow version during PROD image build
 
   -a, --install-airflow-version INSTALL_AIRFLOW_VERSION
-          In CI image, installs Airflow (in entrypoint) from PIP released version or using
-          the installation method specified (sdist, wheel, none).
+          Uses differen version of Airflow when building PROD image.
 
-          In PROD image the installation of selected method or version happens during image building.
-          For PROD image, the 'none' options is not valid.
-
-          One of:
-
-                 2.0.1 2.0.0 1.10.15 1.10.14 1.10.12 1.10.11 1.10.10 1.10.9 none wheel sdist
-
-          When 'none' is used, you can install airflow from local packages. When building image,
-          airflow package should be added to 'docker-context-files' and
-          --install-from-docker-context-files flag should be used. When running an image, airflow
-          package should be added to dist folder and --install-packages-from-dist flag should be used.
+                 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist
 
   -t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
-          If specified, installs Airflow directly from reference in GitHub. This happens at
-          image building time in production image and at container entering time for CI image.
+          Installs Airflow directly from reference in GitHub when building PROD image.
           This can be a GitHub branch like master or v1-10-test, or a tag like 2.0.0a1.
 
   --installation-method INSTALLATION_METHOD
-          Method of installing airflow for production image - either from the sources ('.')
+          Method of installing Airflow in PROD image - either from the sources ('.')
           or from package 'apache-airflow' to install from PyPI.
           Default in Breeze is to install from sources. One of:
 
                  . apache-airflow
 
-  --no-rbac-ui
-          Disables RBAC UI when Airflow 1.10.* is installed.
-
-  --install-packages-from-dist
-          If specified it will look for packages placed in dist folder and it will install the
-          packages after installing Airflow. This is useful for testing provider
-          packages.
-
   --upgrade-to-newer-dependencies
           Upgrades PIP packages to latest versions available without looking at the constraints.
 
   --continue-on-pip-check-failure
           Continue even if 'pip check' fails.
 
+  ****************************************************************************************************
+   Use different Airflow version at runtime in CI image
+
+  --use-airflow-version AIRFLOW_SPECIFICATION
+          In CI image, installs Airflow at runtime from PIP released version or using
+          the installation method specified (sdist, wheel, none). When 'none' is used,
+          airflow is just removed. In this case airflow package should be added to dist folder
+          and --use-packages-from-dist flag should be used.
+
+                 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist none
+
+  --use-packages-from-dist
+          In CI image, if specified it will look for packages placed in dist folder and
+          it will install the packages after entering the image.
+          This is useful for testing provider packages.
+
+  --no-rbac-ui
+          Disables RBAC UI when Airflow 1.10.* is installed.
+
   ****************************************************************************************************
    Credentials
 
@@ -2618,12 +2615,12 @@ This is the current syntax for  `./breeze <./breeze>`_:
           are needed by the extras. When you build image during the development (which is
           default in Breeze) all providers are installed by default from sources.
           You can disable it by adding this flag but then you have to install providers from
-          wheel packages via --install-packages-from-dist flag.
+          wheel packages via --use-packages-from-dist flag.
 
   --disable-pypi-when-building
           Disable installing Airflow from pypi when building. If you use this flag and want
           to install Airflow, you have to install it from packages placed in
-          'docker-context-files' and use --install-from-local-files-when-building flag.
+          'docker-context-files' and use --install-from-docker-context-files flag.
 
   --additional-extras ADDITIONAL_EXTRAS
           Additional extras to pass to build images The default is no additional extras.
@@ -2678,7 +2675,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
   --disable-pip-cache
           Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
 
-  --install-from-local-files-when-building
+  --install-from-docker-context-files
           This flag is used during image building. If it is used additionally to installing
           Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
           in the 'docker-context-files' folder. The same flag can be used during entering the image in

CI.rst

@@ -253,16 +253,6 @@ You can use those variables when you try to reproduce the build locally.
 +-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
 | ``HOST_HOME``                           |             |             |            | Home directory on the host.                     |
 +-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
-|                                                           Image variables                                                          |
-+-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
-| ``INSTALL_AIRFLOW_VERSION``             |             |             |            | Installs Airflow version from PyPI when         |
-|                                         |             |             |            | building image. Can be "none" to skip airflow   |
-|                                         |             |             |            | installation so that it can be installed from   |
-|                                         |             |             |            | locally prepared packages.                      |
-+-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
-| ``INSTALL_AIRFLOW_REFERENCE``           |             |             |            | Installs Airflow version from GitHub            |
-|                                         |             |             |            | branch or tag.                                  |
-+-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
 |                                                      Version suffix variables                                                      |
 +-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
 | ``VERSION_SUFFIX_FOR_PYPI``             |             |             |            | Version suffix used during provider             |

Dockerfile.ci

@@ -260,9 +260,6 @@ ENV AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES}
 ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
 ENV INSTALL_PROVIDERS_FROM_SOURCES=${INSTALL_PROVIDERS_FROM_SOURCES}
 
-ARG INSTALL_FROM_DOCKER_CONTEXT_FILES=""
-ENV INSTALL_FROM_DOCKER_CONTEXT_FILES=${INSTALL_FROM_DOCKER_CONTEXT_FILES}
-
 ARG INSTALL_FROM_PYPI="true"
 ENV INSTALL_FROM_PYPI=${INSTALL_FROM_PYPI}
 
@@ -367,17 +364,6 @@ RUN if [[ ${INSTALL_FROM_PYPI} == "true" ]]; then \
         bash /scripts/docker/install_airflow.sh; \
     fi
 
-# Only copy install_from_docker_context_files.sh to not invalidate cache on other script changes
-COPY scripts/docker/install_from_docker_context_files.sh /scripts/docker/install_from_docker_context_files.sh
-
-# If wheel files are found in /docker-context-files during installation
-# they are also installed additionally to whatever is installed from Airflow.
-COPY docker-context-files/ /docker-context-files/
-
-RUN if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then \
-        bash /scripts/docker/install_from_docker_context_files.sh; \
-    fi
-
 # Copy all the www/ files we need to compile assets. Done as two separate COPY
 # commands so as otherwise it copies the _contents_ of static/ in to www/
 COPY airflow/www/webpack.config.js ${AIRFLOW_SOURCES}/airflow/www/