Fix TestDeleteInactiveChannels flakiness

[prathyushpv]

What changed?

Fix TestDeleteInactiveChannels flakiness.
We read the current time before the despatch loop and resue it. Added a small wait in the test for the dispatch loop to finish after advancing the time.
This will make sure that a dispatch loop has finished before next set of tasks are processed. Otherwise dispatch loop might set old lastActive time for channels.

Why?

How did you test it?

Potential risks

Documentation

Is hotfix candidate?

No

[gow]

How difficult would it be to wire the fake clock all the way to the dispatch loop?

[prathyushpv]

We have the time source inserted to the dispatch loop. But it may read the old value and enter the loop. If this happens, LastUpdatedTime will be the old time for all tasks processed in the loop.

[yycptt]

is the goal here to make sure the dispatch loop can see the advanced time from the eventTimeSource? If so, use a s.Eventually to wait for Now() to be advanced before submitting any new tasks?

[prathyushpv]

The issue is a little different:

Consider that a thread has executed this line:

temporal/common/tasks/interleaved_weighted_round_robin.go

Line 370 in 7bfdbff

now := s.ts.Now()

and entered LoopDispatch below that.

At this exact time, we update time source. ts.Now() is updated. We might add new tasks now. But the above thread has read the old time value already. These new task's LastActiveTime will be updated to the old time value that is already read.

Because of this, these channels have an old LastUpdatedTime and it will be cleaned up when we advance time in test. We have to wait for the execution that read the old time has finished.

One another way to fix this is to run ts.Now() on line 375 for each task.

[yycptt]

Ah I see. Thanks for the explanation.
yeah that's pretty hard to get rid of. Maybe add more context to the comment, essentially saying we have no way to know if a dispatch loop finished or not.