Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core/rawdb: introduce flush offset in freezer #30392

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

rjl493456442
Copy link
Member

@rjl493456442 rjl493456442 commented Sep 4, 2024

It's a following PR of #29792 to get rid of the data file sync.

This is a non-backward compatible change, which increments the database version from 8 to 9.

This pull request introduces a flushOffset for each freezer table, which tracks the position of the most recently fsync’d item in the index file.

When this offset moves forward, it indicates that all index entries below it, along with their corresponding data items, have been properly persisted to disk. The offset can also be moved backward when truncating from either the head or tail of the file.

Previously, the data file required an explicit fsync after every mutation, which was highly inefficient. With the introduction of the flush offset, the synchronization strategy becomes more flexible, allowing the freezer to sync every 30 seconds instead.

The data items above the flush offset are regarded volatile and callers must to ensure they are recoverable after the unclean shutdown, or explicitly sync the freezer before any proceeding operations.

@rjl493456442 rjl493456442 force-pushed the freezer-index-validation-optimize branch 6 times, most recently from 6ed7a57 to ddefb09 Compare September 13, 2024 12:11
@rjl493456442 rjl493456442 force-pushed the freezer-index-validation-optimize branch 4 times, most recently from 28b2490 to 7b3d9bf Compare October 11, 2024 06:38
@rjl493456442 rjl493456442 changed the title [WIP] Freezer index validation optimize core/rawdb: introduce flush offset in freezer Oct 11, 2024
@rjl493456442 rjl493456442 force-pushed the freezer-index-validation-optimize branch from 7b3d9bf to fa72fb3 Compare October 14, 2024 05:56
@rjl493456442 rjl493456442 force-pushed the freezer-index-validation-optimize branch from fa72fb3 to 47da6db Compare October 25, 2024 06:18
@rjl493456442 rjl493456442 force-pushed the freezer-index-validation-optimize branch 2 times, most recently from 00aad3a to 00a251a Compare October 25, 2024 06:47
@rjl493456442 rjl493456442 force-pushed the freezer-index-validation-optimize branch from 00a251a to 5d89190 Compare October 25, 2024 07:06
@fjl
Copy link
Contributor

fjl commented Nov 5, 2024

This needs to be deployed on benchmark machine another time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants