Use modin.pandas when using LocalDaskExecutor

[Sinha-Ujjawal]

Hi,
Can I use modin.pandas inside a task, when running the flow using LocalDaskExecutor?

[zanieb]

Hi! Using a library that schedules distributed work within a task that is also running on that executor is a bit complicated so I don't have a super clear answer. What would you be using as the modin backend? It may be worth taking a look at #3022 to start. Generally, you may be able to get this working locally but run into problems once you try to scale up to something like the distributed DaskExecutor.

Note: From the modin docs at https://modin.readthedocs.io/en/latest/developer/architecture.html it requires Dask Futures which are a Dask Distributed feature so you'd likely need a separate distributed dask client / cluster within your task for this to work within the LocalDaskExecutor