Compatibility issues with polars: Any calculation of polars out of a task will block calculation in the task

[WiC-htao]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.10.0

What happened?

When I create a constant out of tasks using some polars calculation like df.unique or df.with_columns, any polars calculation in tasks will be stuck. The tasks including it will show running and never end.

my polars version is 0.20.19

What you think should happen instead?

The task should run as expected

How to reproduce

from airflow.decorators import task
from airflow.models.dag import DAG
import polars as pl
import pendulum    
S1_TS = pl.DataFrame({"ticktime": [1, 2, 2, 3]}).unique()

with DAG(
    f"test",
    schedule="00 20 * * 1-5",
    start_date=pendulum.datetime(2024, 6, 18),
) as dag:
    # S1_TS = pl.DataFrame({"ticktime": [1, 2, 2, 3]}).unique() # created constant in the dag context will have the same stucks
    @task
    def test():
        x = pl.DataFrame({"a": [1, 2]}).filter(a=1)
    test()

Operating System

Ubuntu 20.04.6 LTS (Focal Fossa)

Versions of Apache Airflow Providers

Deployment

Official Apache Airflow Helm Chart

Deployment details

apache-airflow-providers-cncf-kubernetes 8.3.0
apache-airflow-providers-common-compat 1.2.0
apache-airflow-providers-common-io 1.3.1
apache-airflow-providers-common-sql 1.13.0
apache-airflow-providers-fab 1.1.0
apache-airflow-providers-ftp 3.9.0
apache-airflow-providers-http 4.11.0
apache-airflow-providers-imap 3.6.0
apache-airflow-providers-mysql 5.6.1
apache-airflow-providers-smtp 1.7.0
apache-airflow-providers-sqlite 3.8.0

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

This is an interesting issue but likely environmental discussion. In order for anyone to be able to help to solve your problem I think you need to provide way more details - ideally to some analysis and what polars processess are doing, what your environment is, maybe some logging that indicates what's going on - attaching with debuggers etc.

There is absolutely no indication here that your problem is related to airflow, and not to your environment, and generally - it's very unlikely it has anything to do with Airflow. But if you can dig deeper and provide more information and analysis, then maybe someone will be able to help you.

[potiuk]

Converted to discussion untill there are some evidences this is an airflow issue.

[potiuk]

BTW. some random similar issues returned by google threads mention that polars might hang where it requires to open many sockets/files and they are limited - maybe that is your case on your machine where workers are run (i.e. environmental issue) , but that's a totally wild guess.

[potiuk]

Also another interesting thing https://docs.pola.rs/user-guide/misc/multiprocessing/ -> you might want to look at starting your Polars context with spawn. Airlfow uses heavily multiprocessing and forks and it seems that Polars has specific guidelines and instructions that you should follow on this and some guidelines suggesting to use spawn when forking is involved.. This is a polars doc to follow, not Airflow.