Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] Decouple create worker vs pop worker request. #47694

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

rynewang
Copy link
Contributor

Now, when you call PopWorker(), it finds an idle one or creates a worker. If a new worker is created, the worker is associated to the request and can only be used by it.

This PR decouples the worker creation and the worker-to-task assignment, by adding an abstraction namely PopWorkerRequest. Now, if a req triggers a worker creation, the req is put into a queue. If there are workers ready, that is a PushWorker is called, either from a newly started worker or a released worker, Ray matches the first fitting request in the queue. This reduces latency.

Later it can also be used to pre-start workers more meaningfully.

@rynewang
Copy link
Contributor Author

@jjyao pls take a look at the last commit to see if it makes sense.

@rynewang rynewang assigned rynewang and unassigned jjyao Sep 17, 2024
Signed-off-by: Ruiyang Wang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants