CNDB-12425: A few reproduction tests and a preliminary patch, WIP #1529

ekaterinadimitrova2 · 2025-01-23T03:19:50Z

Queries that could use an index will fail while the index is being built, even if allow filtering is specified.
This is an availability issue for people who migrate from using ALLOW FILTERING to indexes.

...

What does this PR fix and why was it fixed
...
We enable CC to fall back to ALLOW FILTERING on the initial index build, which is considered safe. This is done by adding allowFiltering information (whether it exists in the query and whether the query is supported with ALLOW FILTERING) to the RowFilter and also new Index.Status - INITIAL_BUILD_STARTED
In CNDB, later rebuilds are also safe as the index is queryable while the compactor is rebuilding.
This is acknowledged in CC patch as we check not only that an index is building and we have ALLOW FILTERING, but also that the index is not queryable. The difference between Astra and CC is that in CC index is always not queryable during building.
While I still have to address additional testing for index build during bootstrapping and 2i testing of the patch in CC, that does not matter for Astra, so I believe the patch can be reviewed in parallel. I added two tests as per my conversation with @jasonstack, and they pass. Please let me know if there are any other cases they may need to address and whether the tests are what they had to be.

For the addition of allowFiltering, I had to bump the messaging version.
The migration tests failed but I believe this will be addressed with https://github.com/riptano/cndb/pull/13095. It is solving the issues from bumping the messaging version in CC. It is safe to ignore them for now.
Everything else seems to have passed.

Still missing additional 2i testing and testing of builds during bootstrapping. Also, I want to add a feature flag.

Checklist before you submit for review

Make sure there is a PR in the CNDB project updating the Converged Cassandra version
Use NoSpamLogger for log lines that may appear frequently in the logs
Verify test results on Butler
Test coverage for new/modified code is > 80%
Proper code formatting
Proper title for each commit staring with the project-issue number, like CNDB-1234
Each commit has a meaningful description
Each commit is not very long and contains related changes
Renames, moves and reformatting are in distinct commits

src/java/org/apache/cassandra/db/ReadCommand.java

src/java/org/apache/cassandra/index/SecondaryIndexManager.java

ekaterinadimitrova2 · 2025-02-10T15:06:03Z

test/unit/org/apache/cassandra/service/reads/range/RangeCommandsTest.java

@@ -77,7 +76,7 @@ public void tesConcurrencyFactor()
        // verify that a low concurrency factor is not capped by the max concurrency factor
        PartitionRangeReadCommand command = command(cfs, 50, 50);
        try (RangeCommandIterator partitions = RangeCommands.rangeCommandIterator(command, ONE, System.nanoTime(), ReadTracker.NOOP);
-             ReplicaPlanIterator ranges = new ReplicaPlanIterator(command.dataRange().keyRange(), command.indexQueryPlan(), keyspace, ONE))
+             ReplicaPlanIterator ranges = new ReplicaPlanIterator(command.dataRange().keyRange(), command.indexQueryPlan(), keyspace, ONE, false))


This should be command.rowFilter().allowFiltering

src/java/org/apache/cassandra/db/ReadCommand.java

adelapena · 2025-02-11T15:15:37Z

src/java/org/apache/cassandra/index/SecondaryIndexManager.java

@@ -362,13 +365,24 @@ public synchronized Future<?> addIndex(IndexMetadata indexDef, boolean isNewCF)
     * @param queryPlan a query plan
     * @throws IndexNotAvailableException if the query plan has any index that is not queryable
     */
-    public void checkQueryability(Index.QueryPlan queryPlan)
+    public boolean isQueryableThroughIndex(Index.QueryPlan queryPlan, boolean allowsFiltering)


This method can now return true or false, or throw an exception, and it also throws a client warning. That looks like too many side effects for a method starting with boolean is..., which might suggest a simpler behaviour. I would either:
a) Split it into two simpler separate boolean methods to know if all the indexes in the plan are building/queryable, and let ReadCommand#executeLocally do the AF check and throw exceptions and warnings.
b) Transform it into a SecondaryIndexManger#searcherFor(Index.QueryPlan, boolean) method keeping most of it's responsibilities, returning the searcher if it's possible to build it, null if it's building, and exception if it's not queryable.

src/java/org/apache/cassandra/index/SecondaryIndexManager.java

src/java/org/apache/cassandra/locator/ReplicaPlans.java

src/java/org/apache/cassandra/db/filter/RowFilter.java

src/java/org/apache/cassandra/index/sai/StorageAttachedIndexGroup.java

adelapena · 2025-02-13T17:03:00Z

src/java/org/apache/cassandra/index/Index.java

+         * @return a new query plan for the specified {@link RowFilter} and {@link Index}, {@code null} otherwise
+         */
+        @Nullable
+        QueryPlan queryPlanForIndices(RowFilter rowFilter, Set<Index> indexes);


I'm still thinking about how could make this the only method and get rid of queryPlanFor(RowFilter)

I haven't addressed this yet, I will come back to it soon

src/java/org/apache/cassandra/index/SingletonIndexGroup.java

src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java

src/java/org/apache/cassandra/index/SecondaryIndexManager.java

ekaterinadimitrova2 · 2025-03-03T08:25:15Z

test/distributed/org/apache/cassandra/distributed/test/sai/IndexAvailabilityTest.java

+    }
+
+    public void testAllowFilteringDuringIndexBuildsOn3NodeCluster(boolean isCreateIndex, Index.Status buildStatus) throws Exception
+    {


This is long and ugly, but covers all cases. I will refactor it soon. It wasn't a priority for now

ekaterinadimitrova2 · 2025-03-03T08:25:43Z

test/unit/org/apache/cassandra/cql3/validation/entities/SecondaryIndexTest.java

@@ -1156,6 +1156,8 @@ public void testIndexQueriesWithIndexNotReady()
        {
            execute("DROP index " + KEYSPACE + ".testIndex");
        }
+
+        execute("SELECT value FROM %s WHERE value = 2 ALLOW FILTERING");


Technically this is the only non-sai test.... we need more

test/unit/org/apache/cassandra/db/filter/ANNOptionsTest.java

ekaterinadimitrova2 · 2025-03-03T08:26:43Z

test/unit/org/apache/cassandra/index/AllIndexImplementationsTest.java

+        final Injections.Barrier blockIndexBuild = Injections.newBarrier("block_index_build", 2, false)
+                                                                     .add(InvokePointBuilder.newInvokePoint().onClass(StorageAttachedIndex.class)
+                                                                                            .onMethod("startInitialBuild"))
+                                                                     .build();


I was planning to test here also 2i, but in practice this tests only SAI.... for now...

test/unit/org/apache/cassandra/index/sai/cql/AllowFilteringTest.java

ekaterinadimitrova2 · 2025-03-03T08:32:25Z

Still not ready for full review...
Missing tests and work around bootstrap. I would appreciate any help with ideas for testing in that area and 2i.
I need to add a feature flag and do a full pass to clean the code. I bumped the messaging version, but I believe Michael has some fixes in his branch. I need to pull them on rebase after he merges tomorrow.
I left some comments to mark things that come to my mind that need to be taken care of. Let's also see what CI has to say. Leaving the PR in draft, though, to show it is not the latest version yet.

sonarqubecloud · 2025-03-03T09:11:10Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
90.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

cassci-bot · 2025-03-03T09:15:25Z

✔️ Build ds-cassandra-pr-gate/PR-1529 approved by Butler

Approved by Butler
See build details here

src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java

src/java/org/apache/cassandra/db/ReadCommand.java

adelapena · 2025-03-03T13:37:06Z

src/java/org/apache/cassandra/index/SecondaryIndexManager.java

+
+                // if the status of the index is building and there is allow filtering - that is ok too
+                if (considerAllowFiltering && status == Index.Status.INITIALIZED && allowFiltering)
+                    continue;


Have to think about it more thoroughly, but this looks like a good place to place the client warnings that are currently thrown on the replica side. We might have a warning message per index-building replica, so clients can know what nodes are still initializing their indexes and are going to use filtering.

I added:

// if the status of the index is building and there is allow filtering - that is ok too
if (considerAllowFiltering && status == Index.Status.INITIAL_BUILD_STARTED && !index.isQueryable(status) && allowFiltering)
{
ClientWarn.instance.warn(String.format("Query fell back to ALLOW FILTERING because index %s is still building on endpoint %s",
index.getIndexMetadata().name,
replica.endpoint()));
continue;
}

which led to multiple warnings for the same node in tests.

I decided to just bring on single node C* and try single query on index build:

cqlsh:k> CREATE CUSTOM INDEX ON t(k) USING 'StorageAttachedIndex';
cqlsh:k> SELECT * FROM t WHERE k=200 ALLOW FILTERING;

pk | i | j | k | vec ----+---+---+---+----- (0 rows) Warnings : Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000 Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoint localhost/127.0.0.1:7000

I have to dig into this tomorrow.... no more energy today

I suspect that's because of virtual nodes, with 16 tokens per node. Rather than throwing the client warning immediately, the endpoints can be collected in a set:

Set<InetAddressAndPort> filteringEndpoints = new HashSet<>();

and then throw a single warning after the loop with the unique addresses. For example:

Query fell back to ALLOW FILTERING because index t_k_idx is still building on endpoints 192.168.0.1:7000, 192.168.0.2:7000

I was wondering whether fell back to ALLOW FILTERING will be clear enough for users, considering that they have just written ALLOW FILTERING in the query and, strictly, ALLOW FILTERING is a permission to filter and not the action of filtering. Perhaps the message would be a bit clearer this way:

The query won't use the indexes a, b and c on endpoints 192.168.0.1:7000, 192.168.0.2:7000 because the indexes are still building on those nodes.

Feel free to ignore if you don't agree; I'm just giving ideas.

...org/apache/cassandra/distributed/test/sai/AllowFilteringDuringIndexBuildDistributedTest.java

test/distributed/org/apache/cassandra/distributed/test/ByteBuddyUtils.java

test/distributed/org/apache/cassandra/distributed/test/index/IndexTestBase.java

…t include: - feature flag - checks we are on the new messaging version added for ANNOptions - we fall back to allow filtering only on Index Creation. Currently we also fall back to ALLOW FILTERING if we use nodetool to rebuild indexes

…ull rebuilds Added some ugly testing to IndexAvailabilityTest to confirm queries with the two build statuses

…ordinator and know the plan may change when it reaches the replica and we rebuild it. This will change with CNDB-13129 Added new IndexBuildDuringBootstrapTest. Handle bootstrapping if we think we should? Address other nits and fixes. Rebased on top of Michael's messaging version bump and related fixes.

Split to prevent timeouts. Also add cluster sharing to speed it up.

Fix AllIndexImplementationsTest and extend it to cover other index implementations.

ekaterinadimitrova2 · 2025-03-28T01:54:10Z

test/unit/org/apache/cassandra/index/AllIndexImplementationsTest.java

        if (requiresFiltering)
            assertInvalidThrowMessage(error, InvalidRequestException.class, query);
+        else if (duringInitialBuild)
+            assertInvalidThrow(IndexNotAvailableException.class, query);


@adelapena , after I rebased on top of CNDB-12620, and I realized that no matter what exception I put here, the tests with all other indexes but SAI always pass...

I think we don't really need to do the build injections for anything but SAI for two reasons:

all indexes but SAI are always queryable according to Index.isQueryable . Though it seems during the build at least SASI does not return results - https://github.com/riptano/cndb/issues/12931

I think there was a bug in the test and we get actually An index involved in this query does not support disjunctive queries using the OR operator from the first query once I fix it. We were never hitting the if (duringInitialBuild) with non-SAI indexes

ekaterinadimitrova2 marked this pull request as draft January 23, 2025 03:20

ekaterinadimitrova2 self-assigned this Jan 23, 2025

ekaterinadimitrova2 commented Jan 23, 2025

View reviewed changes

src/java/org/apache/cassandra/db/ReadCommand.java Show resolved Hide resolved

ekaterinadimitrova2 force-pushed the CNDB-12425 branch from 9b73538 to e03373a Compare February 9, 2025 17:51

ekaterinadimitrova2 marked this pull request as ready for review February 9, 2025 17:53

ekaterinadimitrova2 marked this pull request as draft February 9, 2025 17:53

ekaterinadimitrova2 marked this pull request as ready for review February 9, 2025 17:55

ekaterinadimitrova2 marked this pull request as draft February 9, 2025 17:57

ekaterinadimitrova2 marked this pull request as ready for review February 9, 2025 18:01

ekaterinadimitrova2 marked this pull request as draft February 9, 2025 21:10

ekaterinadimitrova2 commented Feb 10, 2025

View reviewed changes

src/java/org/apache/cassandra/index/SecondaryIndexManager.java Outdated Show resolved Hide resolved

ekaterinadimitrova2 commented Feb 10, 2025

View reviewed changes

src/java/org/apache/cassandra/db/ReadCommand.java Outdated Show resolved Hide resolved

adelapena reviewed Feb 11, 2025

View reviewed changes

ekaterinadimitrova2 requested a review from michaeljmarshall February 11, 2025 18:53

adelapena reviewed Feb 13, 2025

View reviewed changes

ekaterinadimitrova2 force-pushed the CNDB-12425 branch from 7138378 to 8e0677f Compare March 3, 2025 08:18

ekaterinadimitrova2 commented Mar 3, 2025

View reviewed changes

src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java Outdated Show resolved Hide resolved

ekaterinadimitrova2 commented Mar 3, 2025

View reviewed changes

src/java/org/apache/cassandra/index/SecondaryIndexManager.java Show resolved Hide resolved

ekaterinadimitrova2 commented Mar 3, 2025

View reviewed changes

src/java/org/apache/cassandra/index/SecondaryIndexManager.java Outdated Show resolved Hide resolved

ekaterinadimitrova2 commented Mar 3, 2025

View reviewed changes

test/unit/org/apache/cassandra/db/filter/ANNOptionsTest.java Outdated Show resolved Hide resolved

ekaterinadimitrova2 commented Mar 3, 2025

View reviewed changes

test/unit/org/apache/cassandra/index/sai/cql/AllowFilteringTest.java Outdated Show resolved Hide resolved

ekaterinadimitrova2 marked this pull request as ready for review March 3, 2025 08:28

ekaterinadimitrova2 marked this pull request as draft March 3, 2025 08:32

adelapena reviewed Mar 3, 2025

View reviewed changes

ekaterinadimitrova2 force-pushed the CNDB-12425 branch from 8e0677f to 552a7b6 Compare March 4, 2025 01:55

ekaterinadimitrova2 commented Mar 4, 2025

View reviewed changes

test/distributed/org/apache/cassandra/distributed/test/ByteBuddyUtils.java Outdated Show resolved Hide resolved

ekaterinadimitrova2 commented Mar 4, 2025

View reviewed changes

test/distributed/org/apache/cassandra/distributed/test/ByteBuddyUtils.java Show resolved Hide resolved

ekaterinadimitrova2 commented Mar 4, 2025

View reviewed changes

test/distributed/org/apache/cassandra/distributed/test/index/IndexTestBase.java Outdated Show resolved Hide resolved

ekaterinadimitrova2 commented Mar 4, 2025

View reviewed changes

test/distributed/org/apache/cassandra/distributed/test/index/IndexTestBase.java Outdated Show resolved Hide resolved

ekaterinadimitrova2 added 11 commits March 15, 2025 18:22

Empty-Commit

3bd67f7

Make testAllowFilteringDuringIndexBuildsOn3NodeCluster deterministic

3e5d0ba

A few tiny fixes and spring cleaning

273b51f

Preliminary MV bump, need to check Michael's PR and apply fixes

8d18cb1

Differentiate between initial build on user index creation and next f…

5ed5baf

…ull rebuilds Added some ugly testing to IndexAvailabilityTest to confirm queries with the two build statuses

Fixes needed after rebase

a54967a

Cleaning and adding additional queryability check for Astra

a086490

Address review comments

66331ce

Small fixes around the messaging version bump

6202b6d

ekaterinadimitrova2 force-pushed the CNDB-12425 branch from 552a7b6 to 8c95eb6 Compare March 16, 2025 18:19

sbtourist and others added 3 commits March 19, 2025 16:27

Moved Gossiper check after peerIndexStatus is populated.

9dbd6ae

Split AllowFilteringDuringIndexBuildDistributedTest

5abaa29

Split to prevent timeouts. Also add cluster sharing to speed it up.

Fix and extend AllIndexImplementationsTest

e2ea9dc

Fix AllIndexImplementationsTest and extend it to cover other index implementations.

adelapena force-pushed the CNDB-12425 branch from 411dbb6 to e2ea9dc Compare March 19, 2025 16:29

ekaterinadimitrova2 commented Mar 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CNDB-12425: A few reproduction tests and a preliminary patch, WIP #1529

CNDB-12425: A few reproduction tests and a preliminary patch, WIP #1529

ekaterinadimitrova2 commented Jan 23, 2025 •

edited

Loading

ekaterinadimitrova2 Feb 10, 2025

adelapena Feb 11, 2025

adelapena Feb 13, 2025

ekaterinadimitrova2 Mar 3, 2025

ekaterinadimitrova2 Mar 3, 2025

ekaterinadimitrova2 Mar 3, 2025

ekaterinadimitrova2 Mar 3, 2025 •

edited

Loading

ekaterinadimitrova2 commented Mar 3, 2025

sonarqubecloud bot commented Mar 3, 2025

cassci-bot commented Mar 3, 2025

adelapena Mar 3, 2025

ekaterinadimitrova2 Mar 4, 2025

adelapena Mar 4, 2025

adelapena Mar 4, 2025

ekaterinadimitrova2 Mar 28, 2025

ekaterinadimitrova2 Mar 28, 2025

CNDB-12425: A few reproduction tests and a preliminary patch, WIP #1529

Are you sure you want to change the base?

CNDB-12425: A few reproduction tests and a preliminary patch, WIP #1529

Conversation

ekaterinadimitrova2 commented Jan 23, 2025 • edited Loading

Checklist before you submit for review

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ekaterinadimitrova2 Mar 3, 2025 • edited Loading

Choose a reason for hiding this comment

ekaterinadimitrova2 commented Mar 3, 2025

sonarqubecloud bot commented Mar 3, 2025

Quality Gate passed

cassci-bot commented Mar 3, 2025

✔️ Build ds-cassandra-pr-gate/PR-1529 approved by Butler

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ekaterinadimitrova2 commented Jan 23, 2025 •

edited

Loading

ekaterinadimitrova2 Mar 3, 2025 •

edited

Loading