Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use cluster default remote store path type during snapshot restore #12753

Conversation

ashking94
Copy link
Member

@ashking94 ashking94 commented Mar 19, 2024

Description

This PR is a sub task of feature request #12567 and being tracked in meta issue - #12589.

In the prior PR #12607, we have introduced remote store path type in the customData field in IndexMetadata. We associate the path during index creation. However, there are certain cases where the association needs to be handled manually. They are as follows -

  1. Snapshot restore - For indexes that were backed up in earlier versions (2.12 or earlier), the custom data would be absent. In such cases, we need to associate the newly restored indexes with the cluster default cluster.remote_store.index.path.prefix.type to benefit from the optimised prefix. In absence of this PR, we will continue to use non-optimised prefix.
  2. Migration of indexes from Document replication type to Segment backed by remote store.

In this PR, we are handling 1st case.

Related Issues

Resolves #12730

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • [ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • [ ] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Storage:Performance Storage:Resiliency Issues and PRs related to the storage resiliency v2.14.0 labels Mar 19, 2024
Copy link
Contributor

github-actions bot commented Mar 19, 2024

Compatibility status:

Checks if related components are compatible with change 45aaf5b

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/performance-analyzer.git]

Copy link
Contributor

❕ Gradle check result for 3f2262e: UNSTABLE

  • TEST FAILURES:
      2 org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests.classMethod
      1 org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests.testResizeQueueDown

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Copy link

codecov bot commented Mar 19, 2024

Codecov Report

Attention: Patch coverage is 91.66667% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 71.35%. Comparing base (b15cb0c) to head (45aaf5b).
Report is 57 commits behind head on main.

Files Patch % Lines
...h/cluster/metadata/MetadataCreateIndexService.java 90.90% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #12753      +/-   ##
============================================
- Coverage     71.42%   71.35%   -0.07%     
- Complexity    59978    60153     +175     
============================================
  Files          4985     4999      +14     
  Lines        282275   283045     +770     
  Branches      40946    41030      +84     
============================================
+ Hits         201603   201958     +355     
- Misses        63999    64377     +378     
- Partials      16673    16710      +37     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ashking94
Copy link
Member Author

Ran the MetadataCreateIndexServiceTests.testRemoteCustomData for around 1K iterations after the change and they are successful.

@ashking94 ashking94 force-pushed the remote-path-default-snapshot-restore branch from 3f2262e to 08cb85c Compare March 20, 2024 03:57
Copy link
Contributor

✅ Gradle check result for 08cb85c: SUCCESS

Copy link
Contributor

❕ Gradle check result for f8f6806: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.indices.replication.SegmentReplicationAllocationIT.testSingleIndexShardAllocation

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Ashish Singh <[email protected]>
@ashking94 ashking94 force-pushed the remote-path-default-snapshot-restore branch from f8f6806 to 45aaf5b Compare March 20, 2024 09:15
@ashking94
Copy link
Member Author

❕ Gradle check result for f8f6806: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.indices.replication.SegmentReplicationAllocationIT.testSingleIndexShardAllocation

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Flaky test - #11299

Copy link
Contributor

✅ Gradle check result for 45aaf5b: SUCCESS

@ashking94
Copy link
Member Author

tagging @gbbafna, @linuxpi, @sachinpkale for review.

@sachinpkale sachinpkale merged commit 0510c5b into opensearch-project:main Mar 20, 2024
122 of 131 checks passed
ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Apr 23, 2024
…pensearch-project#12753)

* Use cluster default remote store path type as fallback during snapshot restore

---------

Signed-off-by: Ashish Singh <[email protected]>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…pensearch-project#12753)

* Use cluster default remote store path type as fallback during snapshot restore

---------

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
@ashking94 ashking94 added the backport 2.x Backport to 2.x branch label Apr 26, 2024
@ashking94 ashking94 deleted the remote-path-default-snapshot-restore branch April 26, 2024 05:04
opensearch-trigger-bot bot pushed a commit that referenced this pull request Apr 26, 2024
…12753)

* Use cluster default remote store path type as fallback during snapshot restore

---------

Signed-off-by: Ashish Singh <[email protected]>
(cherry picked from commit 0510c5b)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
sachinpkale pushed a commit that referenced this pull request Apr 26, 2024
…12753) (#13401)

* Use cluster default remote store path type as fallback during snapshot restore

---------


(cherry picked from commit 0510c5b)

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch enhancement Enhancement or improvement to existing feature or request skip-changelog Storage:Performance Storage:Resiliency Issues and PRs related to the storage resiliency v2.14.0
Projects
Status: ✅ Done
Development

Successfully merging this pull request may close these issues.

[Remote Store] Use cluster default path type during snapshot restore
3 participants