Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable https reduction (off NGINX server only) and auto-detect storage_type #245

Open
wants to merge 24 commits into
base: pyfive
Choose a base branch
from

Conversation

valeriupredoi
Copy link
Collaborator

@valeriupredoi valeriupredoi commented Mar 10, 2025

Description

Yet another contribution towards #231

  • introduce a utility that should be able to detect storage type, and enable the user not to be forced to specify the storage_type kwarg in Active
  • allow for files and datasets from files on https servers to be handled, with the caveat that reductions happen inside the PyAS client, and not yet delegated to Reductionist

Reduction inside PyAS client is obviously slower than if delegated to Reductionist, but it's still a good exercise, and it scales very well (ie 2x data = 2x time)

Before you get started

Checklist

  • This pull request has a descriptive title and labels
  • This pull request has a minimal description (most was discussed in the issue, but a two-liner description is still desirable)
  • Unit tests have been added (if codecov test fails)
  • All tests pass

@valeriupredoi valeriupredoi added the enhancement New feature or request label Mar 11, 2025
@valeriupredoi valeriupredoi changed the title Enable https reduction (off NGINX server only) Enable https reduction (off NGINX server only) and auto-detect storage_type Mar 11, 2025
print("Gateway protocol indicates S3 storage.")
return "s3"
else:
return "https"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bnlawrence this is the bit I was on about on the call

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's great. I think we'd need to put the response to that in the json going to reductionist, rather than in the URI itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants