Airflow executing task's two time randomly

[aboratravix]

Hi Team,

We see that airflow randomly executes the same task twice with no clear explanation or pattern. Below is some context on our setup:

1. We have hybrid executors: Local Executor and Kubernetes Executor
2. We have a Postgres DB, and we use PgBouncer to establish connections.
3. We have replica=2 for the scheduler. (https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/local.html)

Attached are two logs of same task where at 12:15 it ran twice, and at 14:15 it ran once correctly. This happens randomly for other pipelines as well.
We till now see this situation only when the task executes as a Local Executor.

dag_id=load_marketing_cost_of_acquisition-COA_run_id=scheduled__2024-09-14T12_15_00+00_00_task_id=upload_BUDGETAIRUS_to_gcs_attempt=1.log
dag_id=load_marketing_cost_of_acquisition-COA_run_id=scheduled__2024-09-14T14_15_00+00_00_task_id=upload_BUDGETAIRUS_to_gcs_attempt=1.log