-
Notifications
You must be signed in to change notification settings - Fork 110
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix GPU error messages for fuzzy deduplication #387
base: main
Are you sure you want to change the base?
Conversation
sarahyurick
commented
Nov 22, 2024
•
edited
Loading
edited
- Raise a clear and readable error if the Dask client is not GPU-based
- Raise a clear and readable error if the dataset’s backend is not cuDF
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
cc @Maghoumi please let me know if these error messages look more how you would expect them. |
Signed-off-by: Sarah Yurick <[email protected]>
@@ -124,6 +124,12 @@ def buckets_to_edges( | |||
return result_df | |||
|
|||
def __call__(self, dataset: DocumentDataset) -> DocumentDataset: | |||
if "cudf" not in str(type(dataset.df)): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this pattern is repeated, it might be better to move to a separate helper (e.g. checks.py
) so it can be re-used in different places.
Something like: ensure_cudf(dataset, module="buckets_to_edges")
which would print something like:
A Dask DataFrame with cuDF backend is required to run 'buckets_to_edges'. Please convert using....
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, I see what you mean but for now I'm not sure whether it's worth adding a new script for this. I will ask others to review as well, thank you.