Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Remote Compaction] Parse options from OPTIONS file instead of using CompactionServiceInput #13025

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

jaykorean
Copy link
Contributor

@jaykorean jaykorean commented Sep 20, 2024

Summary

We've been serializing and deserializing DBOptions and CFOptions (and other CF into) as part of CompactionServiceInput. These are all readily available in the OPTIONS file and the remote worker can read the OPTIONS file to obtain the same information. This helps reducing the size of payload significantly.

The only downside of this approach is that if fail_if_options_file_error is false, we cannot safely guarantee that OPTIONS file is up-to-date with DB's current state. fail_if_options_file_error option is marked for deprecation. We will consider fail_if_options_file_error=false with remote compaction as incompatible features.

This also solves the problem where we had to open the default CF with the CFOption from another CF if the remote compaction is for a non-default column family. (TODO comment in /db_impl_secondary.cc)

Test Plan

Unit Tests

./compaction_service_test
./compaction_job_test

Also tested with Meta's internal Offload Infra

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants