Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DocDB] [DB Clone] Clone is failing with catalog snapshot invalidated, MISMATCHED_SCHEMA #25929

Closed
1 task done
dhruva4869 opened this issue Feb 7, 2025 · 0 comments
Closed
1 task done
Assignees
Labels
area/docdb YugabyteDB core features kind/bug This issue is a bug priority/medium Medium priority issue status/awaiting-triage Issue awaiting triage

Comments

@dhruva4869
Copy link

dhruva4869 commented Feb 7, 2025

Jira Link: DB-15245

Description

Description:
Version: 2.25.1.0-b280

Steps:

  1. Create tables, indexes (unique, partial, covering, expression, secondary), materialized views, FK and partitions.
  2. Load data in all of them and create snapshot schedule
  3. Validate snapshot schedule and take time t1
  4. Do bunch of alters, delete some data and truncate some table
  5. Drop some indexes, Create new indexes (on new column)
  6. Load some data and take time t2
  7. Clone to time t1

Clone doesn't go through with following Error:

[
    {
        "aggregate_state": "ABORTED",
        "source_namespace_id": "00004000000030008000000000000000",
        "seq_no": "1",
        "target_namespace_name": "db4",
        "restore_time": "2025-02-07 10:53:25.031021",
        "abort_message": "Runtime error (yb/util/subprocess.cc:632): Subprocess '/home/yugabyte/yb-software/yugabyte-2.25.1.0-b280-centos-x86_64/bin/../postgres/bin/ysql_dump' terminated with non-zero exit status 256: ysql_dump: error: query failed: ERROR:  The catalog snapshot used for this transaction has been invalidated: expected: 62, got: 37: MISMATCHED_SCHEMA\nysql_dump: detail: Query was: SELECT last_value, is_called FROM public.dummy_table_i2_seq\n"
    }
]

Minimal repro thanks to @yamen-haddad

CREATE DATABASE db4;
\q
./build/latest/bin/yb-admin --master_addresses 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 create_snapshot_schedule 2 320 ysql.db4
./bin/ysqlsh -d db4

CREATE TABLE dummy_table(id text , i2 serial, name text);

SELECT (EXTRACT (EPOCH FROM CURRENT_TIMESTAMP)*1000000)::decimal(38,0);
-- t1= 1738966318840176

ALTER TABLE dummy_table RENAME COLUMN name TO text_column2;

CREATE DATABASE db5 TEMPLATE db4 AS OF 1738966318840176;

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.
@dhruva4869 dhruva4869 added area/docdb YugabyteDB core features status/awaiting-triage Issue awaiting triage labels Feb 7, 2025
@yugabyte-ci yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue labels Feb 7, 2025
yamen-haddad added a commit that referenced this issue Feb 19, 2025
…as of time

Summary:
Currently, clone is failing when the database has a sequence, and the time we are cloning to is a time before a DDL that increments the last_breaking_version of the source database. Clone is failing in ysql_dump phase when executing the following query to get the sequence data: `SELECT last_value, is_called FROM public.seq_name`. The error is:
 `ERROR:  The catalog snapshot used for this transaction has been invalidated: expected: 2, got: 1: MISMATCHED_SCHEMA`.

Generally, ysql_dump sets the GUC `yb_disable_catalog_version_check` to true, which should not set the catalog version in all the read requests that ysql_dump is issuing, and thus no catalog version check is performed.
However, as sequences have their own read path, the GUC `yb_disable_catalog_version_check` is not taking effect, and the catalog version check is returning a catalog version mismatch.

Fixed by extending the role of `yb_disable_catalog_version_check` to cover the sequences' read path. When set to true, we don't send the catalog version with the read request to the tserver and thus disable the catalog version check for such a sequence read query. This allows ysql_dump to read the sequence as of a time regardless of the catalog version at that time and thus, being able to clone sequences to a time before a DDL happened.
Jira: DB-15245

Test Plan: ./yb_build.sh --cxx-test integration-tests_minicluster-snapshot-test --gtest_filter Colocation/PgCloneTestWithColocatedDBParam.CloneWithSequencesAndDdl/0

Reviewers: asrivastava

Reviewed By: asrivastava

Subscribers: ybase, yql, slingam

Differential Revision: https://phorge.dev.yugabyte.com/D41911
vaibhav-yb pushed a commit to vaibhav-yb/yugabyte-db that referenced this issue Feb 20, 2025
…quences as of time

Summary:
Currently, clone is failing when the database has a sequence, and the time we are cloning to is a time before a DDL that increments the last_breaking_version of the source database. Clone is failing in ysql_dump phase when executing the following query to get the sequence data: `SELECT last_value, is_called FROM public.seq_name`. The error is:
 `ERROR:  The catalog snapshot used for this transaction has been invalidated: expected: 2, got: 1: MISMATCHED_SCHEMA`.

Generally, ysql_dump sets the GUC `yb_disable_catalog_version_check` to true, which should not set the catalog version in all the read requests that ysql_dump is issuing, and thus no catalog version check is performed.
However, as sequences have their own read path, the GUC `yb_disable_catalog_version_check` is not taking effect, and the catalog version check is returning a catalog version mismatch.

Fixed by extending the role of `yb_disable_catalog_version_check` to cover the sequences' read path. When set to true, we don't send the catalog version with the read request to the tserver and thus disable the catalog version check for such a sequence read query. This allows ysql_dump to read the sequence as of a time regardless of the catalog version at that time and thus, being able to clone sequences to a time before a DDL happened.
Jira: DB-15245

Test Plan: ./yb_build.sh --cxx-test integration-tests_minicluster-snapshot-test --gtest_filter Colocation/PgCloneTestWithColocatedDBParam.CloneWithSequencesAndDdl/0

Reviewers: asrivastava

Reviewed By: asrivastava

Subscribers: ybase, yql, slingam

Differential Revision: https://phorge.dev.yugabyte.com/D41911
yamen-haddad added a commit that referenced this issue Mar 24, 2025
…eading sequences as of time

Summary:
Original commit: 7fc910d / D41911
Currently, clone is failing when the database has a sequence, and the time we are cloning to is a time before a DDL that increments the last_breaking_version of the source database. Clone is failing in ysql_dump phase when executing the following query to get the sequence data: `SELECT last_value, is_called FROM public.seq_name`. The error is:
 `ERROR:  The catalog snapshot used for this transaction has been invalidated: expected: 2, got: 1: MISMATCHED_SCHEMA`.

Generally, ysql_dump sets the GUC `yb_disable_catalog_version_check` to true, which should not set the catalog version in all the read requests that ysql_dump is issuing, and thus no catalog version check is performed.
However, as sequences have their own read path, the GUC `yb_disable_catalog_version_check` is not taking effect, and the catalog version check is returning a catalog version mismatch.

Fixed by extending the role of `yb_disable_catalog_version_check` to cover the sequences' read path. When set to true, we don't send the catalog version with the read request to the tserver and thus disable the catalog version check for such a sequence read query. This allows ysql_dump to read the sequence as of a time regardless of the catalog version at that time and thus, being able to clone sequences to a time before a DDL happened.
Jira: DB-15245

Test Plan: ./yb_build.sh --cxx-test integration-tests_minicluster-snapshot-test --gtest_filter Colocation/PgCloneTestWithColocatedDBParam.CloneWithSequencesAndDdl/0

Reviewers: asrivastava

Reviewed By: asrivastava

Subscribers: slingam, yql, ybase

Differential Revision: https://phorge.dev.yugabyte.com/D42696
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docdb YugabyteDB core features kind/bug This issue is a bug priority/medium Medium priority issue status/awaiting-triage Issue awaiting triage
Projects
None yet
Development

No branches or pull requests

3 participants