-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[core][dashboard] Change file_tail_iterator to async. #47721
Conversation
Signed-off-by: Ruiyang Wang <[email protected]>
Signed-off-by: Ruiyang Wang <[email protected]>
Signed-off-by: Ruiyang Wang <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The change makes sense to me, but I’m curious why Ray 2.9 works. I thought we’ve been using Iterator instead of AsyncIterator for years.
Signed-off-by: Ruiyang Wang <[email protected]>
didn't take time to blame out the problematic pr... |
Signed-off-by: Ruiyang Wang <[email protected]>
@kevin85421 so it's #44658 from 5 mo ago I think |
Hi @rynewang @kevin85421 will this fix be included in the ray nightly release? thanks! |
) On `ray job submit`, the CLI tails logs from the job agent. The agent needs to read log tails from an iterator and yields to a WebSocket. However the file reading iterator is SYNC so it blocks agent event loop, causing the agent to block, making downstream consumers like KubeRay to break. Changes the log reading function `file_tail_iterator` to an AsyncIterator and a 1s `time.sleep` to `asyncio.sleep` to unblock. Signed-off-by: Ruiyang Wang <[email protected]> Signed-off-by: ujjawal-khare <[email protected]>
On
ray job submit
, the CLI tails logs from the job agent. The agent needs to read log tails from an iterator and yields to a WebSocket. However the file reading iterator is SYNC so it blocks agent event loop, causing the agent to block, making downstream consumers like KubeRay to break. Changes the log reading functionfile_tail_iterator
to an AsyncIterator and a 1stime.sleep
toasyncio.sleep
to unblock.The issue was introduced from #44658.
Fixes #47637.
Fixes ray-project/kuberay#2355