Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include dask-cloudprovider? (Maybe?) #513

Open
thwllms opened this issue Feb 8, 2024 · 1 comment
Open

Include dask-cloudprovider? (Maybe?) #513

thwllms opened this issue Feb 8, 2024 · 1 comment

Comments

@thwllms
Copy link

thwllms commented Feb 8, 2024

Is your feature request related to a problem? Please describe.
I'm not totally clear on how most folks are deploying Dask clusters recently, but I've found dask-cloudprovider very helpful with that and have been including it in project images that use pangeo/pangeo-notebook as a base.

Describe the solution you'd like
Does it make sense to include dask-cloudprovider as a dependency in pangeo-notebook/environment.yml?

Describe alternatives you've considered
🤷‍♂️

Additional context
If I'm missing something and there's a better way to stand up Dask clusters with Fargate/etc., I'd love to hear it. (Does everyone just use Coiled these days?) If this has been considered already and the decision has been to leave dask-cloudprovider out of the Pangeo images because the images are meant to be more about compute environment rather than infra, I could understand that choice.

@scottyhq
Copy link
Member

scottyhq commented Apr 2, 2024

If this has been considered already and the decision has been to leave dask-cloudprovider out of the Pangeo images because the images are meant to be more about compute environment rather than infra

This is certainly true, but I think we're open to any suggestions for making the images easy to use on various infrastructure setups and documentation is always welcome! Ideally the infrastructure would not require heavy dependencies in the user environment (I'm not sure whatdask-cloudprovider requires nowadays but imagine it's similar to dask-gateway and coiled packages)

Generally, though, images defined in this repository have focused on being used on JupyterHubs with Dask-Gateway. If you have other needs the recommended approach has been fork the repository and customize the images to your needs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants