Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] Please provide better message where 'RuntimeError: Failed to unpickle serialized exception' #49885

Open
Joshuaalbert opened this issue Jan 16, 2025 · 2 comments
Labels
bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component)

Comments

@Joshuaalbert
Copy link

What happened + What you expected to happen

You could at least tell us what line it's occuring on, and also likely where the origin of the exception is.

(DataStreamer pid=974272) Failed to unpickle serialized exception
(DataStreamer pid=974272) Traceback (most recent call last):
(DataStreamer pid=974272)   File "/usr/local/lib/python3.11/site-packages/ray/exceptions.py", line 51, in from_ray_exception
(DataStreamer pid=974272)     return pickle.loads(ray_exception.serialized_exception)
(DataStreamer pid=974272)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(DataStreamer pid=974272)   File "/usr/local/lib/python3.11/site-packages/jax/_src/errors.py", line 314, in __init__
(DataStreamer pid=974272)     f"{tracer._error_repr()}{tracer._origin_msg()}")
(DataStreamer pid=974272)        ^^^^^^^^^^^^^^^^^^
(DataStreamer pid=974272) AttributeError: 'str' object has no attribute '_error_repr'
(DataStreamer pid=974272) 
(DataStreamer pid=974272) The above exception was the direct cause of the following exception:
(DataStreamer pid=974272) 
(DataStreamer pid=974272) Traceback (most recent call last):
(DataStreamer pid=974272)   File "/usr/local/lib/python3.11/site-packages/ray/_private/serialization.py", line 460, in deserialize_objects
(DataStreamer pid=974272)     obj = self._deserialize_object(data, metadata, object_ref)
(DataStreamer pid=974272)           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(DataStreamer pid=974272)   File "/usr/local/lib/python3.11/site-packages/ray/_private/serialization.py", line 342, in _deserialize_object
(DataStreamer pid=974272)     return RayError.from_bytes(obj)
(DataStreamer pid=974272)            ^^^^^^^^^^^^^^^^^^^^^^^^
(DataStreamer pid=974272)   File "/usr/local/lib/python3.11/site-packages/ray/exceptions.py", line 45, in from_bytes
(DataStreamer pid=974272)     return RayError.from_ray_exception(ray_exception)
(DataStreamer pid=974272)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(DataStreamer pid=974272)   File "/usr/local/lib/python3.11/site-packages/ray/exceptions.py", line 54, in from_ray_exception
(DataStreamer pid=974272)     raise RuntimeError(msg) from e
(DataStreamer pid=974272) RuntimeError: Failed to unpickle serialized exception

Versions / Dependencies

2.39

Reproduction script

Not possible because the poor exception messages makes it hard to find out where the problem is.

Issue Severity

High: It blocks me from completing my task.

@Joshuaalbert Joshuaalbert added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Jan 16, 2025
@wingkitlee0
Copy link
Contributor

I saw some jax code there. How are you calling the remote function and jax array?

@Joshuaalbert
Copy link
Author

@wingkitlee0 I am able to find the source of the problem after a trial and error search with debug messages since JAX is used in many places. However, I'm loath to discuss the cause of the exception (which is simple to solve once found) because I've been asking Ray to put better messages in the case of unserialisable exceptions for some time, and their engineers like to ignore the request and talk rather about the exception. It's an edge case that an exception is not serialisable, and they should handle it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component)
Projects
None yet
Development

No branches or pull requests

2 participants