Some tasks in mapped tasks fail before starting, logs are empty

[michaelimas]

Hi dear community,
Will appreciate your insight on this issue we are experiencing-

Once in a while, we experience failed task instances without any log, as if they never started but still marked as failed.

This is how the log of one of the tasks look like:

This happens only in mapped tasks and only for some of the tasks.
When clearing the DAG run it succeeds so it seems like a temporary issue.
It doesn't trigger the on_failure_callback so we don't even get an indication which makes it a very high severity issue.

Looking at the metrics in datadog i see that the scheduler pods have terminated around that time and 2 new pods were initialised ~5 mins beforehand, probably due to deployment.

My questions are:

1. Which component is responsible for this "fault"- is it the scheduler? the worker? on which level these tasks got stuck
2. What can cause that component to "drop" these tasks? maybe you have an idea on the relevant configuration that we can tune.

Any help will be very appriciated.
We run a Kubenetes executor and Airflow v2.9.3

[enchant3dmango]

Could you share the DAG code and the code you are triggering?

Also, try to check the related log in your S3 bucket.