Fix misbehaving tests

[cgtz]

(Edits by @vgkholla to delete tests that are no longer relevant)
The following tests were failing intermittently or hanging and were commented out in PR #294. This ticket tracks their fixes.

In BlockingChannelConnectionPoolTest:

• testBlockingChannelInfoForPlainText (fails intermittently)

com.github.ambry.network.BlockingChannelConnectionPoolTest > testBlockingChannelInfoForPlainText FAILED 
java.lang.AssertionError at BlockingChannelConnectionPoolTest.java:242

• testSSLBlockingChannelConnectionPool (fails intermittently)

com.github.ambry.network.BlockingChannelConnectionPoolTest > testSSLBlockingChannelConnectionPool FAILED
java.lang.Exception at BlockingChannelConnectionPoolTest.java:396

In SimpleByteBufferPoolTest:

• testOneExpiredAnotherServed (hangs intermittently, seemingly deadlocked)

one thread:
at java.lang.Thread.join(Thread.java:1354) 
at com.github.ambry.utils.SimpleByteBufferPoolTest.testOneExpiredAnotherServed(SimpleByteBufferPoolTest.java:220)

another thread:
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
at com.github.ambry.utils.SimpleByteBufferPoolTest$BufferConsumer.run(SimpleByteBufferPoolTest.java:259)

[sriramsub]

Tagged with newbie. Might be useful for someone new to debug the tests and learn the code.

[greplog]

@sriramsub Hey, I am exploring the testRollingIndex test, can you give me the scenario for reproducing the test failure?

[pnarayanan]

@greplog This is indeed a good starter issue to look into.

The test fails intermittently. @cgtz may have more idea about how to reproduce it. But when it does fail, we see the following:

com.github.ambry.store.PersistentIndexTest > testRollingIndex FAILED
java.lang.AssertionError at PersistentIndexTest.java:841
java.lang.NullPointerException: at com.github.ambry.store.PersistentIndexTest.testRollingIndex(PersistentIndexTest.java:837)

This is most likely an issue with the test being incorrect. Either way, I would recommend starting with these steps:

1. Get a basic understanding of what the PersistentIndex class is meant for and what all it does by going through the class here: ambry-store/src/main/java/com.github.ambry.store/PersistentIndex.java.
2. Go through the test to see what it is trying to test and if it is doing it the right way, if there are any obvious issues that you can see.
3. Enable testRollingIndex() test by uncommenting the @Test in this file: ambry-store/src/test/java/com.github.ambry.store/PersistentIndexTest.java

Feel free to ask us about anything at [email protected].

[greplog]

@pnarayanan Thanks for the heads up and error log, as you suggested, I am going through the code, and will try to reproduce and fix it.

[cgtz]

Hi @greplog! To help with reproducing this bug, after uncommenting the test you want to target, try running the tests in a loop using this bash command:

count=0; while (./gradlew clean && ./gradlew build && [ $count -ne 50 ]); do ((count++)); printf "\n\nDone with %d iterations\n\n" $count; done

[greplog]

Hi all, there are other tests also failing in the current system like DiskTest.testDiskSoftState

Plus, can somebody help me with the difference in task unitTest and test, I am assuming test is running integration plus unit tests, while unitTest is running only unit tests.

stacktrace of other failing tests
com.github.ambry.clustermap.DiskTest > testDiskSoftState FAILED
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:145)
at com.github.ambry.clustermap.DiskTest.testDiskSoftState(DiskTest.java:142)

26 tests completed, 1 failed
:ambry-clustermap:unitTest FAILED

[cgtz]

Hi @greplog, thanks for finding this! What type of system did it fail on? I can add it to the list.

Regarding the test and unitTest targets, you are correct. test runs the server integration tests as well. We set it up this way so that the build target will just run the unit test suite but test will run both.

[greplog]

Hi @cgtz
I ran the test on mac OSX Yosemite, with Java Environment 1.8, Disk : SSD

[vgkholla]

testBlockingChannelInfoForSSL also failed recently

com.github.ambry.network.BlockingChannelConnectionPoolTest > testBlockingChannelInfoForSSL FAILED
    java.lang.AssertionError: expected:<1> but was:<2>
        at org.junit.Assert.fail(Assert.java:91)
        at org.junit.Assert.failNotEquals(Assert.java:645)
        at org.junit.Assert.assertEquals(Assert.java:126)
        at org.junit.Assert.assertEquals(Assert.java:470)
        at org.junit.Assert.assertEquals(Assert.java:454)
        at com.github.ambry.network.BlockingChannelConnectionPoolTest.underSubscriptionTest(BlockingChannelConnectionPoolTest.java:310)
        at com.github.ambry.network.BlockingChannelConnectionPoolTest.testBlockingChannelInfo(BlockingChannelConnectionPoolTest.java:238)
        at com.github.ambry.network.BlockingChannelConnectionPoolTest.testBlockingChannelInfoForSSL(BlockingChannelConnectionPoolTest.java:171)

attn: @nsivabalan