Fix [MQB]: mqbc::StorageMgr: Transition to available only when all primary active

[kaikulimu]

Fixes two flaky integration tests in FSM mode:

1. In test_basic of test_restart.py, there was an issue where a replica could advertise availability before all primaries are active; then, a proxy could repoen queue and post message with no avail. The fix is to transition to available only when all primaries are active.

2. In test_kill_post_start of test_strong_consistency.py, there was an issue where replicas are not issuing receipts to the primary after restart. This was because healing replicas in FSM mode were not buffering primary status advisories to process later, and thus not setting the correct primary in the FileStore. After I added the buffering logic, the tests pass.

[dorjesinpo]

Couple questions...

[dorjesinpo]

why is this in setPrimaryStatusForPartition and not in setPrimaryForPartition?

[kaikulimu]

FileStore should only set primary after the primary becomes active. The FileStore is unable to work with a passive primary. I will rename the function as FileStore::setActivePrimary to make the point clear.

[dorjesinpo]

Question, does pinfo.primaryNode() get assigned upon PrimaryStatusAdvisory or in the FSM flow there is another trigger? We receive PrimaryStatusAdvisory and we do not have partition primary, why not assign the primary then?

[kaikulimu]

In FSM mode, the source of truth for partition assignments are in the cluster state snapshot of the CSL file. As part of healing, a new leader assigns partitions and then applies the assignments in its first CSL advisory. Primary status adviosries can be stale; that's why we have a lot of checks in this function in the first place. My original idea was to simply ignore all primary status advisories and purely rely upon FSM for partition assignments. However, FSM can heal a replica but neglect to set a primary as active. Thus, I came up with the idea of buffering primary status advisories. If an advisory is not stale (i.e. matching primary node and leaseId), then we trust the availability advisory.

[bmq-oss-ci]

Build 250 of commit e5d59b2 has completed with FAILURE

[bmq-oss-ci]

Build 251 of commit 0116bdf has completed with FAILURE

[kaikulimu]

@dorjesinpo Back to you

[bmq-oss-ci]

Build 256 of commit 337db61 has completed with FAILURE

[bmq-oss-ci]

Build 266 of commit 8059ba6 has completed with FAILURE

[dorjesinpo]

one question

[dorjesinpo]

Do we need to

  if(allParitionsAvailable()) {
            d_recoveryStatusCb(0);

or is it done implicitly and if so, maybe comment should explain that

[kaikulimu]

Will add a comment explaining how d_recoveryStatusCb is called a bit later.

[kaikulimu]

What if you get a series of ACTIVE, then PASSIVE, then ACTIVE buffered advisories? How are you going to process them?

[kaikulimu]

Currently, we do not do anything special when a primary becomes ACTIVE -> PASSIVE -> ACTIVE. Thus, we also do no-op here. Arguably, we can improve the logic, but it's beyond the scope of this PR.

[bmq-oss-ci]

Build 297 of commit 9a258b4 has completed with FAILURE

[bmq-oss-ci]

Build 301 of commit f6f34d4 has completed with FAILURE