Use patch file to use requests to download dataset for 24.12 #723
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
raydouglass
reviewed
Dec 17, 2024
There was a problem hiding this comment.
If going this route, this patch file needs to be copied into the container, like here: https://github.com/rapidsai/docker/blob/branch-25.02/cuvs-bench/gpu/Dockerfile#L55
raydouglass
reviewed
Dec 17, 2024
| import subprocess | ||
| import sys | ||
| -from urllib.request import urlretrieve | ||
| +import requests |
There was a problem hiding this comment.
Does the container have requests installed?
raydouglass
approved these changes
Dec 19, 2024
bdice
reviewed
Jan 6, 2025
There was a problem hiding this comment.
Why patch this if we can make the fix directly in cuvs-bench?
edit: Got the answer offline. 24.12 cuvs-bench artifacts were already published, which necessitated a patch. This change could be made for 25.02, but we are going to try and self-host the datasets to avoid this problem entirely.
rapids-bot bot
pushed a commit
that referenced
this pull request
Jan 15, 2025
Separates the workflows for building the RAPIDS end user images and the cuVS images. The cuVS images do not depend on the RAPIDS end user images, so they can be built in parallel. This also allows for finer grained retries in case of failures. Also switches to using `rapids-mamba-retry` for installing conda packages. Finally, disables building the `cuvs-bench-datasets` images which are consistently failing (#724) until a better solution than the workaround in #723 is ready. 5adab54 can be reverted to re-enable this. Authors: - Ray Douglass (https://github.com/raydouglass) Approvers: - Jake Awe (https://github.com/AyodeAwe) URL: #725
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is to work around the 403 issue being seen when downloading datasets, while we create a more permanent solution.