Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT MERGE][Object Store HA][ProtoType] Object Store HA #27541

Closed
wants to merge 39 commits into from
Closed

[DO NOT MERGE][Object Store HA][ProtoType] Object Store HA #27541

wants to merge 39 commits into from

Conversation

Catch-Bull
Copy link
Contributor

@Catch-Bull Catch-Bull commented Aug 5, 2022 •
edited
Loading

Why are these changes needed?

This is a prototype test of Object Store HA. REP: ray-project/enhancements#10
The object which supports HA we called HA_Object:

  • To reduce workload, we use spill/restore objects to implement dump/load checkpoints. In the final PR, we still tend to implement dump/load checkpoints alone on the basis of using the existing code as much as possible.
  • Although we call PinObjects when we seal HA_Object, HA_Object doesn't have the primary copy, it is only so that it can be spilled. When the global owner restarts and the global references are released, there is no good idea on how to delete the primary copy.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

kfstorm and others added 30 commits May 19, 2022 19:33
@kfstorm
Metadata and schema
82bfb51
@Catch-Bull
save
0295dc9
@Catch-Bull
[core] Fix Object Manager owner address after AssignOwner (ray-proj…
4ab7c0c 
…ect#25021)

When assigning an owner for an object (different from the current worker), such as:
```python
ray.put(vaule, _owner = ACTORHANDLE)
```
Object Manager holds the wrong owner's address and updates location info to the wrong worker, making `ray.get` slow. the current master will get Timeout in this new test case.
@kfstorm
Update global owner code
3905181
@Catch-Bull
save
10b41c8
@Catch-Bull
save
3448114
@Catch-Bull
save
65ba9c7
@Catch-Bull
Merge branch 'object_store_ha' of github.com:alipay/ray into object_s…
11730e2 
…tore_ha_backup
@Catch-Bull
save
40c8d88
@Catch-Bull
handle location
b823e5d
@Catch-Bull
erase item in objects_need_to_report_
cb8da6b
@kfstorm
Remove dump/load checkpoint (part1)
674e3b3
@kfstorm
Update JNI
607e810
@kfstorm
Update object manager
2bae536
@kfstorm
rename checkpoint URL to spilled URL
9d32476
@kfstorm
update
17ee75e
@kfstorm
pass test_owner_failed
96bb8f9
@kfstorm
update test
8f4a177
@Catch-Bull
save
9e7d43e
@Catch-Bull
add object location
aae9836
@Catch-Bull
save
ad6771d
@Catch-Bull
save
55d804a
@Catch-Bull
lint
5019d8d
@Catch-Bull
add _ha
456c48e
@Catch-Bull
delete subscribe on CoreWoker
5a5e825
@Catch-Bull
lint
4c9b25a
@Catch-Bull
fix
0a198d4
@Catch-Bull
save
f7c849f
@Catch-Bull
fix
b974a8b
@Catch-Bull
Merge remote-tracking branch 'opensource/master' into object_store_ha…
3ec7f77 
…_clean
@kfstorm
Copy link
Member

kfstorm commented Aug 5, 2022

The current prototype is:

  • An HA object is spilled on creation. Since it's spilled to remote storage, it doesn't have a primary copy. All copies in plasma are secondary copies.
  • We assume that we don't GC data on remote storage, even if an HA object is out-of-scope, so we don't care about reference counting either. Global owners are only responsible for location recording to avoid unnecessary restoring from remote storage. It doesn't need to manage references.

@stephanie-wang stephanie-wang self-assigned this Aug 9, 2022
@stephanie-wang
Copy link
Contributor

Will try to review this soon, but we're a bit busy at the moment with the 2.0 release.

@ericl ericl self-assigned this Aug 9, 2022
@ericl
Copy link
Contributor

ericl commented Aug 9, 2022

This is very exciting. Also looking forwards to reviewing after 2.0.

@Catch-Bull Catch-Bull mentioned this pull request Aug 19, 2022
Open
@stephanie-wang
Copy link
Contributor

Thanks for the prototype! A few things:

  • It seems like it would simplify the code a lot if we make actor_id and checkpoint_url optional fields of the object's ownership info
  • I found it a bit hard to follow all the pieces of code since there is a lot of boilerplate code as well. Can you summarize the major changes that were made to each component?
  • How do we delete HA objects / GC the metadata? It seems like there could be a situation where we end up losing all the language-level copies of the ObjectRef, so no one can call free, but the system still thinks that there is such an object. Can we rethink the API to support this case?
  • How will it work for objects created by .remote()?

@Catch-Bull
Copy link
Contributor Author

Thanks for the prototype! A few things:

  • It seems like it would simplify the code a lot if we make actor_id and checkpoint_url optional fields of the object's ownership info
  • I found it a bit hard to follow all the pieces of code since there is a lot of boilerplate code as well. Can you summarize the major changes that were made to each component?
  • How do we delete HA objects / GC the metadata? It seems like there could be a situation where we end up losing all the language-level copies of the ObjectRef, so no one can call free, but the system still thinks that there is such an object. Can we rethink the API to support this case?
  • How will it work for objects created by .remote()?
  • Thanks for the reminder, I will try it on the next PR.
  • no problem, I will also implement HA-returned-object in the prototype later, and I will summarize it together later.
  • Since HA-Object no longer has a primary copy, HA-Object's metadata no longer has the reference count, but only contains location information. It is indeed possible that HA-Object's reference does not exist, but its data in plasma is not deleted, and the location still exists, but I think this "leak" is ok.
  • I will implement HA-returned-object in this prototype later.

@stale
Copy link

stale bot commented Oct 1, 2022

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

  • If you'd like to keep this open, just leave any comment, and the stale label will be removed.

@stale stale bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Oct 1, 2022
@ericl ericl assigned fishbone, ericl and scv119 and unassigned ericl and fishbone Oct 3, 2022
@stale
Copy link

stale bot commented Oct 18, 2022

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!

@stale stale bot closed this Oct 18, 2022
@Catch-Bull Catch-Bull reopened this Nov 16, 2022
@stale stale bot removed the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Nov 16, 2022
@Catch-Bull Catch-Bull changed the title [Object Store HA][ProtoType] Object Store HA [DO NOT MERGE][Object Store HA][ProtoType] Object Store HA Nov 16, 2022
@stale
Copy link

stale bot commented Dec 17, 2022

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

  • If you'd like to keep this open, just leave any comment, and the stale label will be removed.

@stale stale bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Dec 17, 2022
@stale
Copy link

stale bot commented Dec 31, 2022

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!

@stale stale bot closed this Dec 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wuisawesome wuisawesome Awaiting requested review from wuisawesome

@ericl ericl Awaiting requested review from ericl

@AmeerHajAli AmeerHajAli Awaiting requested review from AmeerHajAli

@robertnishihara robertnishihara Awaiting requested review from robertnishihara

@pcmoritz pcmoritz Awaiting requested review from pcmoritz pcmoritz is a code owner

@raulchen raulchen Awaiting requested review from raulchen raulchen is a code owner

@fishbone fishbone Awaiting requested review from fishbone

@scv119 scv119 Awaiting requested review from scv119

@stephanie-wang stephanie-wang Awaiting requested review from stephanie-wang

@kfstorm kfstorm Awaiting requested review from kfstorm

Assignees

@ericl ericl

@scv119 scv119

@stephanie-wang stephanie-wang

Labels
stale The issue is stale. It will be closed within 7 days unless there are further conversation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@Catch-Bull @kfstorm @stephanie-wang @ericl @scv119 @fishbone