Get KIF CI working reliably with Xcode 14.3.1 & 15.4 #1300
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
justinseanmartin
force-pushed
the
jmartin/xcode-15
branch
2 times, most recently
from
b283126
to
cdf27e4
Compare
justinseanmartin
force-pushed
the
jmartin/xcode-15
branch
from
a776837
to
b9b5426
Compare
justinseanmartin
force-pushed
the
jmartin/xcode-15
branch
15 times, most recently
from
b1b1145
to
d279cd7
Compare
justinseanmartin
force-pushed
the
jmartin/xcode-15
branch
from
d279cd7
to
f170a50
Compare
justinseanmartin
changed the title
Configuring CI to run in 4 configurations: - xcode 14.3.1, iPad Air (5th generation), iOS 16.4 - xcode 14.3.1, iPhone 14, iOS 16.4 - xcode 15.4, iPad Air (5th generation), iOS 17.5 - xcode 15.4, iPhone 15, iOS 17.5 As part of this change, the CI script was updated to only run the extra validation of the examples folder one one of the 4 shards. This removes Carthage from the README as being officially supported. I'd guess that is generally not being used. The CI script hasn't worked since Xcode 12.
* AccessibilityIdentifierTests tests Fix `testEnteringTextIntoViewWithAccessibilityIdentifier` flaking We're trying to tap on `Select All` while it is still being animated into view. The problem is two-fold: 1. The `longPress` action doesn't wait for animations to complete, it just uses a hardcoded .5s before starting the next action. 2. Even after switching to use `waitForAnimationsToFinish`, the timeout defaults to .5s as the max to wait for animations to complete. To fix this, I switched `longPress` action to use `waitForAnimationsToFinish` and I upped the default timeout to wait up to 2s for animations to complete. This seems to work much more reliably now. * SystemTests.testMockingOpenURL NSURL no longer parses '423543523454://', use a different protocol that starts with characters * ExistTests Wait a bit longer for the view controller presentation to complete * TypingTests In TapViewController, avoid race condition in resigning first responder and becoming first responder. CI was flaking with a failure where "Deleted something., world" was found in the greeting text field. * PullToRefreshTests Leverage 'usingCurrentFrame' so we're not scrolling the table view around while the UIRefreshControl is spinning, which immediately halts the refresh spinner. Also, make sure that we're scrolled to the top of the test suite VC, as otherwise the pull to refresh action might fail.
It doesn't really make sense that the default animation wait time is the same as the stabilization time. This effectively means the behavior of scanning the UI hierarchy for animations in progress is never utilized for most tests out of the box. For what its worth, we've used a 2s animation wait time and .1s stabilization time as our defaults for a very long time. This should generally not slow down any tests if there weren't any animations in progress. If there were animations in progress, this is probably a worthwhile tradeoff to make things more reliable. This can still be configured globally to be overidden. As part of this change, use `waitForAnimationsToFinish` in place of a static .5s sleep for the `longPress` action. There will be a followup PR kif-framework#1301 that should speed up some scenarios where we're waiting longer than desired for animations to complete.
Have the pullToRefresh action pull from 25% of the way down the element to 75% of the way down, instead of pulling from the center and pulling a full height of the element
This removes the `KIFFramework` target and its consumer tests. They don't interop well with the way that angle and quote imports work in SPM. More info: https://joesusnick.medium.com/swift-package-manager-with-a-mixed-swift-and-objective-c-project-part-2-2-e71dad234e6 Took this opportunity to remove the spaces in the target names for the 'Testable Swift' target, in case that was causing issues somewhere.
justinseanmartin
force-pushed
the
jmartin/xcode-15
branch
3 times, most recently
from
9eb5577
to
615592a
Compare
1 task
justinseanmartin
force-pushed
the
jmartin/xcode-15
branch
from
615592a
to
ecb0c32
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'd recommend reviewing this commit-by-commit, as they're logically organized and have verbose commit messages. Some notable things:
KIF behavior changes:
PullToRefresh test flakiness:
This was probably the trickiest thing that I discovered while digging in here. There were a bunch of flakes happening on CI that I couldn't really explain. What I saw when running these tests locally was the pull to refresh control would be dismissed abruptly after the action completed, despite it being configured to wait for up to 4s to complete. What I found was that the behavior in #1020 to instantly scroll without animations to enumerate table view cells is what is causing the abrupt cancellation of the pull to refresh operation. This was unexpected and I think a side-effect of the behavior change in #1020.
Luckily, in #1296, @SimoncelloCT added the capability to prevent scrolling the view when enumerating the view hierarchy. I leveraged the functionality there to stabilize these tests.
Other test flakiness:
There were a bunch of small changes required to get the tests to pass more reliably. I tried reproducing a lot of these with slow animations enabled locally in the simulator settings. CI is working much better now with this change.
Misc cleanup: