[Bug] Protobuf upgrade breaks MWAA

[MazrimT]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

All versions of dbt-core is not installable in amazon MWAA after pr #10969 was merged.
Reason is that MWAA enforces airflow constraints https://raw.githubusercontent.com/apache/airflow/constraints-2.10.3/constraints-3.11.txt which specifies only a single version of protobuf==4.25.5
Even one step newer airflow 2.10.4 and with python 3.12 still only have that specific version.
This means dbt-core can not install on MWAA.

Expected Behavior

dbt-core (or any adapter using it) should not require such high version of protobuf and at a minimum require only the version that airflow has as constraints.

Steps To Reproduce

Try to install dbt-core (or dbt-snowflake) with the following at the top of requirements.txt file:
--constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.10.3/constraints-3.11.txt

Relevant log output

Environment

- OS:
- Python:
- dbt:

Which database adapter are you using with dbt?

No response

Additional Context

No response

[aleksandrgordienko]

Just observing and, unfortunately, not able to help with the issue itself…

Have you ever considered installation of dbt outside of Airflow? I found it quite difficult to always keep dependencies in sync for Airflow and dbt in one installation. If you split this two from each other, you can upgrade them separately and maintain dependencies separately two

[aranke]

Hi @MazrimT,

There are two issues at play here:

1. Protobuf major version changes are not backwards-compatible
2. Airflow/MWAA is using constraints as a lock file and has pinned to protobuf==4.25.5

Unfortunately this means that we need to keep the protobuf dependency tightly pinned, given how critical it is to end user code, to avoid unexpected breakages (as has happened in the past).

The workaround here is what @aleksandrgordienko mentioned above, keeping dbt and Airflow in two separate environments.

I'm sorry I couldn't be of more help, but I'm going to close this issue for now.

Links to previous discussions in case it's helpful:

1. [CT-1392] [Bug] dbt-core 1.3.0 compatibility issue with apache-airflow 2.2.2 #6128 (comment)
2. Airflow installation recommendation is incorrect aws/aws-mwaa-local-runner#371 (comment)