Skip to content

FAT tests

Jump to bottom
Volodymyr Siedlecki edited this page Nov 29, 2023 · 75 revisions

Table of Contents

Overview

The majority of OpenLiberty tests are Functional Acceptance Tests (FAT). A FAT test project ends in _fat and executes tests with a running OpenLiberty image. This differs from Unit tests because Unit tests do not execute on a running OpenLiberty image.

The example FAT project is build.example_fat and can be used as a minimalistic reference point for writing new FATs.

TCK projects

Note that TCK projects follow a slightly different layout. A TCK project ends in _fat_tck and require mvn to be installed when run locally. See here for more information on creating and running a TCK project.

FAT Project Layout

- com.ibm.ws.my_fat/
  - fat/src/            # Root folder for test client side
    - com/ibm/ws/my/
      - FATSuite.java   # Entry point of the FAT.  Lists one or more test classes via '@SuiteClasses'
      - MyTest.java     # Test class containing a group of tests.  Server and app lifecycle is controlled here.
  - test-applications/  # Root folder for all test applications
    - myApp/            # Folder name indicates name of application (by convention)
      - resources/      # Optional: Static resources such as web.xml are included here for 'myApp'
      - src/            # Java code for 'myApp' goes here
  - publish/            # Root folder for non-java resources
    - files/            # Files needed by test client go here.  Such as alternate server.xml configurations
    - servers/          # Root folder for OpenLiberty servers
      - myServer/       # Folder representing a server.  This folder gets copied to ${wlp.install.dir} at runtime
        - server.xml    # Server configuration for 'myServer'
  - build/libs/autoFVT/ # Root folder for a single execution of FAT output
    - output/           # Root folder for server output
    - results/          # Root folder for test client output
      - junit.html/     # Contains JUnit-style test results which can be viewed in a web browser
      - output.txt      # Log file for test client (i.e. fat/src/ classes)
  - bnd.bnd             # Describes source folders and compile-time build dependencies
  - build.gradle        # Build file.  Must include "apply from: '../cnf/gradle/scripts/fat.gradle'"

How to run an existing FAT

To run a FAT, you must have ant on your $PATH.
To test this, you can run the command which ant. If ant is not on your path, you can download ant here and then add export PATH=$ANT_HOME/bin:$PATH to your ~/.bashrc. After adding ant to your path, run ./gradlew --stop to stop the Gradle daemon so that it can be started with the updated $PATH.

To run an entire FAT, use the command:

./gradlew <fat_project_name>:buildandrun

For example:

./gradlew build.example_fat:buildandrun

The buildandrun task will build and run the FAT. If you are not making changes to the FAT and just want to re-run the FAT without rebuilding it, you can use the runfat task instead.

To run only a single test class in a FAT, use the command:

./gradlew <fat_project_name>:buildandrun -Dfat.test.class.name=<fully_qualified_java_test_class>

For example:

./gradlew build.example_fat:buildandrun -Dfat.test.class.name=com.ibm.ws.example.SimpleTest

You no longer have to fully qualify the test name and can specify multiple test classes with a comma separated list:

./gradlew build.example_fat:buildandrun -Dfat.test.class.name=SimpleTestA,SimpleTestB

You can also use '*' to wildcard match multiple test classes:

./gradlew build.example_fat:buildandrun -Dfat.test.class.name=SimpleTest*

To run only a single test in a FAT class, use the command:

./gradlew <fat_project_name>:buildandrun -Dfat.test.class.name=<fully_qualified_java_test_class> -Dfat.test.method.name=<java_method_name>

For example:

./gradlew build.example_fat:buildandrun -Dfat.test.class.name=com.ibm.ws.example.SimpleTest -Dfat.test.method.name=exampleTest

You can now specify multiple test classes with a comma separated list:

./gradlew build.example_fat:buildandrun -Dfat.test.class.name=SimpleTest -Dfat.test.method.name=exampleTest1,exampleTest2

You can also use '*' to wildcard match multiple tests:

./gradlew build.example_fat:buildandrun -Dfat.test.class.name=SimpleTest -Dfat.test.method.name=exampleTest*

Overview of how a FAT test runs

When a FAT runs there are two or more JVMs in use:

  • A test-client side JVM which manages server lifecycle, deploys apps, and drives individual test cases on the server side.
  • An OpenLiberty server with zero or more test apps running on it.

Most FATs use a test client to drive HTTP GET requests on a test servlet within a test application. The steps for this style of FAT test is the following.

  1. The build packages everything needed to run the FAT into an autoFVT.zip archive.
  2. On an automated build, the autoFVT.zip for each FAT is built on the parent build machine. Then the autoFVT.zip archives are distributed evenly among about a dozen child build engines where they will be extracted and run. This allows us to pack about 36 hours of FAT tests into about 3 hours.
  3. The build unzips autoFVT.zip and starts a JUnit process with the class named FATSuite as the entry point. Each test class in the @SuiteClasses will be run in sequence.
  4. Test class runs
    1. Test class @BeforeClass runs. Typically applications are deployed to server and then server is started here.
    2. Test client runs each test case in class in sequence. Typically each test case is a single HTTP GET. For example:
      GET http://localhost:8010/myApp/MyTestServlet?testMethod=myTest
      1. Test application receives HTTP GET request via a Servlet. For example, inside MyTestServlet.doGet() the testMethod parameter is checked, and reflection is used to invoke a method indicated by the parameter value, so MyTestSevlet.myTest() is invoked.
      2. If the test case passes, the test servlet prints a well-known success message string to the HTTPServletResponse. If the test case fails, the test servlet does not print the success message, and instead dumps the stack trace of the failure to the HTTPServletResponse.
      3. Test client reads HTTPServletResponse. If response contains well-known success message, the test passes. If the success message is missing, fail the test case with the HTTPServletResponse content as the failure message.
    3. Test class @AfterClass runs. Typically server is stopped here. When server is stopped, the server logs are copied from ${wlp.install.dir} into build/libs/autoFVT/output/servers/<server_name>-<stop_timestamp>

Using the fattest.simplicity infrastructure

The FAT test infrastructure is housed in the fattest.simplicity project. This contains all of the junit extensions and extra bells and whistles available to FAT projects.

Enabling the simplicity infrastructure

To get most of the custom value-adds of the simplicity infrastructure, test classes must indicate @RunWith(FATRunner.class) on the test class. For example:

@RunWith(FATRunner.class)
public class SimpleTest { /* ... */ }

Controlling server lifecycle with LibertyServer

The LibertyServer class gives the test client a POJO representation of an OpenLiberty server. The easiest way to obtain an object instance is with the @Server("<server_name>") annotation. With this object, you can do things like start and stop a server.

An example usage typically looks like:

@RunWith(FATRunner.class)
public class MyFATTest {

  @Server("MyServer")
  public static LibertyServer server;

  @BeforeClass
  public static void setup() throws Exception {
    server.startServer();
  }

  @AfterClass
  public static void tearDown() throws Exception {
    server.stopServer();
  }

}

Building and deploying test applications with ShrinkWrap

Typically FATs build application artifacts at FAT runtime using an open source library called ShrinkWrap, and then export the artifacts to disk. The ShrinkWrap API is quite flexible and can build many different types of archives including: .zip .jar .war .ear .rar

The official ShrinkWrap javadoc can be found here: ShrinkWrap API 1.2.3 API

The ShrinkWrap API can get a little verbose, so we have a ShrinkHelper class to make conventional operations more concise. A few useful ones are:

  • ShrinkHelper.defaultApp(LibertyServer s, String appName, String... packages)
    Builds an artifact named <appName>.war which includes resources from test-applications/<appName>/resources/ and the specified java packages. Also exports the .war to the indicated LibertyServer's ${server.config.dir}/apps/ directory.
  • ShrinkHelper.defaultDropinApp(LibertyServer s, String appName, String... packages)
    The same thing as ShrinkHelper#defaultApp except it exports the .war to the ${server.config.dir}/dropins/, which means no server.xml configuration is needed for the application.
  • ShrinkHelper.buildDefaultApp(String appName, String... packages)
    The same thing as ShrinkHelper#defaultApp except the archive is not exported to disk anywhere.
  • ShrinkHelper.exportAppToServer(LibertyServer s, Archive<?> a)
    Exports the archive to disk in the ${server.config.dir}/apps/ directory. The name of the exported file will be a.getName().
  • ShrinkHelper.exportToServer(LibertyServer s, String path, Archive<?> a)
    Same thing as ShrinkHelper#exportAppToServer except you can indicate a path to export the archive at (relative to ${server.config.dir} of the indicated server).
  • ShrinkHelper.addDirectory(Archive<?> a, String path)
    Adds a directory (and all sub-dirs) to the indicated archive.

Server-side JUnit and Servlet template with componentTest-1.0

There is a test-only feature in Liberty called componentTest-1.0 which will provide some common utilities for FAT test apps.

  • If the componentTest-1.0 feature is enabled in server.xml, then JUnit will be made available to test applications. This is useful for being able to do server-side assertions.

  • If the componentTest-1.0 feature is enabled in server.xml, the FATServlet class is made available to test applications. This class contains boiler-plate HttpServelt code for invoking test methods.

Declaring test methods on the server side using @TestServlet

Without @TestServlet
With standard JUnit a "test stub" would be needed on the client side for each server-side test. For example:

// Client side
public class MyTest {

  // get and start server in @BeforeClass

  @Test
  public void myTest() throws Exception {
    // invoke GET http://loclahost:8010/myApp/MyTestServlet?testMethod=myTest and assert response is successful
    FATServletClient.runTest(server, "/myApp/MyTestServlet", "myTest");
  }
}

// Server side
@WebServlet("/MyTestServlet")
public class MyTestServlet extends FATServlet {
  public void myTest() throws Exception {
    // run some server-side test
  }
}

This is inconvenient because the client side code is completely boiler plate and if the server side test moves or is renamed it requires a corresponding update on the client side class.

With @TestServlet
To eliminate this, we have the @TestServlet annotation which will scan the referenced test servlet class for @Test methods and add synthetic client-side test stubs at runtime. So the example above becomes:

// Client side
@RunWith(FATRunner.class)
public class MyTest {
  @Server("MyServer")
  @TestServlet(servlet = MyTestServlet.class, contextRoot = "myApp") // References class gets scanned for @Test methods
  public static LibertyServer server;

  // start server in @BeforeClass
}

// Server side
@WebServlet("/MyTestServlet")
public class MyTestServlet extends FATServlet {
  @Test // Able to use standard JUnit @Test annotation in test servlet
  public void myTest() throws Exception {
    // run some server-side test
  }
}

Debugging a FAT

In a typical FAT there are two or more JVMs (client driver, and a server).

To debug the test client, use the -Ddebug.framework option and attach a debugger on port 6666. For example:

./gradlew build.example_fat:buildandrun -Ddebug.framework

If you are using the Eclipse IDE, you can attach a debugger by selecting the debug configuration Liberty-FAT remote debug from the debug menu dropdown. If you've never used this debug configuration, it will not appear in the dropdown until you've visited Debug Configurations... from the Debug dropdown. Look for 'Liberty-FAT remote debug' under 'Remote Java Application'.

To debug a liberty server, use the -Ddebug.server option and attach a debugger on port 7777. For example:

./gradlew build.example_fat:buildandrun -Ddebug.server

If you are using the Eclipse IDE, you can attach the debugger by selecting Liberty- server remote debug from the debug menu dropdown. See above if this item doesn't appear in your debug dropdown.

FAT test modes

There are two primary FAT test modes: LITE and FULL, with LITE being the default mode.

LITE FULL
Target runtime Under 5 minutes per FAT project Under 30 minutes per FAT project
Covers
  • The most common or ideal scenario, where everything is configured exactly as it needs to be
  • The configurations or scenarios that are most likely to appear in the field, including error path cases
  • long running tests
  • less common situtations / configurations
  • anything else not covered by LITE mode
Run in scheduled builds Many times per day Several times per week
Run in personal builds Whenever a feature used by any of the servers defined in the test is modified Whenever the test project itself is updated

The simplicity test framework has an @Mode annotation for this purpose. If a test method does not define a @Mode annotation, it will be considered a LITE mode test. For example:

public class MyTest {
  @Test
  public void regularTest() { } // No @Mode annotation, so LITE mode is assumed

  @Test
  @Mode(LITE)
  public void liteTest() { } // functionally equivalent to 'regularTest()'

  @Test
  @Mode(FULL)
  public void testThatTakesAWhile() { } // Will only be run the FAT is run in FULL mode
}

To run a FAT in FULL mode, specify -Dfat.test.mode=FULL in the launch command, for example:

./gradlew build.example_fat:buildandrun -Dfat.test.mode=FULL

12/2020 update: Alternatively, it's found this annotation @Mode(TestMode.FULL) at the test class level (instead of above test method level) seems to work well, for example:

@Mode(TestMode.FULL)
@RunWith(FATRunner.class)
public class SampleTests {
. . .

And some other tips learned:

  • Since OL (open-liberty) FAT no longer uses FATSuiteLite.java to specify LITE FAT bucket (as used in CL/WS-CD-Open), we need to ensure the test classes defined for the OL FAT project are clearly specified for which will run as FULL mode and which as LITE mode. If using the annotation @Mode(TestMode.FULL) for FULL mode, then the test classes without this notation will be run as LITE mode.
  • OL Personal Build contains the property fat.test.mode with lite default value, when there is one point we have the entire FAT suite classes annotated with FULL mode (due to wrong assumption and user error), and the build.log results an error "NO TESTS REPORTED IN THE JUNIT REPORT", which took us some time to figure out the root cause.

Tips for writing fast FAT tests

Having FAT tests which run quickly makes our builds run faster and keeps your development round-trip times shorter. However, reaching the target runtimes for FULL and LITE FAT modes can be a challenge, so here are some tips for keeping your test times down.

Make sure you put tests in LITE and FULL mode appropriately

Follow the guidance above to make sure that tests which cover less used scenarios are annotated with @Mode(FULL) so that they don't run in LITE mode.

Avoid restarting the server unnecessarily

Although liberty starts very quickly the time spent waiting for the server to be ready can be one of the slowest parts of a FAT bucket if you do it a lot. To avoid restarting:

  • Minimise the number of different server.xmls your tests use (particularly in LITE mode)
  • Use one test class per server
    • Have the test class deploy test applications and start the server in a @BeforeClass method and stop the server in an @AfterClass method
    • Use @TestServlet multiple times from within this single test class to run the tests from all of your test servlets
    • See FaultToleranceMainTest for an example of a test class which runs almost all of the core Fault Tolerance tests on one server.

Be careful about how much you RepeatTests

Repeating Tests is a great way to ensure we have test coverage for every version of our feature, without having to maintain multiple versions of all of our tests. However, you may not need to repeat everything against every configuration, especially in LITE mode. Use RepeatTestAction.fullModeOnly() on any repeat configurations that only need to run in FULL mode.

Analyse output.txt to see where your time is really going

If you're not sure where your FAT bucket is spending its time, then looking through output.txt can give you a better idea.

output.txt is the log file produced by the FAT framework when it runs your tests. After running your FAT project, it can be found in build/libs/autoFVT/results.

As long as you're using FATServletClient and the @Server annotation, pull out the timestamps for when your build started and stopped the server and when it started and stopped each test with this command:

grep -E '<<<|>>>' output.txt

Look down the list of timestamps for large gaps which indicate something taking a long time. It can also be helpful to count the number of server starts that you're doing and how long they tend to take to see how much of your test time is really taken up by starting the server.

If all else fails, split up your buckets

Some OpenLiberty features are large, contain a lot of functionality and require a lot of tests for adequate coverage. In this case, you may need to split up your tests into multiple FAT projects to meet the target time limits.

Having many small FAT projects is still better than having a few large FAT projects because our builds run FAT projects in parallel, so we can run many small projects much more quickly.

Repeating FAT tests for different feature/JEE levels

When we have multiple versions of a feature (e.g. cdi-1.2 and cdi-2.0), it is useful to re-run the same tests on the newer feature levels (e.g. re-run the CDI 1.2 tests for both cdi-1.2 and cdi-2.0) in order to prove that functionality that worked in previous feature versions continues to work in newer feature versions.

Important note:

The RepeatTests ClassRule described below will repeat tests within the same bucket as the original. This means that a bucket that previously took 5 minutes to run in LITE mode may take 10 minutes or more if repeated in its entirety. This problem is even worse for a FULL mode run which may also run some long and complex tests.

Therefore, it is important to be selective about which tests to repeat and in which modes. It is recommended that only the latest version of a feature should be tested in LITE mode. Other versions should only be repeated in FULL mode. It may not even be necessary to repeat some tests at all, particularly in cases where multiple versions of a feature share a significant amount of common code. You should carefully assess which tests to repeat and in which mode to run them.

For example, if a FAT was written with Java EE 7 features, and you want to re-run the FAT with Java EE 8 equivalent features, you can do the following:

@RunWith(Suite.class)
@SuiteClasses({ ... })
public class FATSuite {
    // Using the RepeatTests @ClassRule in FATSuite will cause all tests in the FAT to be run twice.
    // First without any modifications, then again with all features in all server.xml's upgraded to their EE8 equivalents.
    @ClassRule
    public static RepeatTests r = RepeatTests.withoutModification()
                    .andWith(FeatureReplacementAction.EE8_FEATURES());
}

Note that when using the FeatureReplacementAction.EE8_FEATURES action, the additional test iteration will be skipped automatically if the java level is less than Java 8.

For a more fine-grained approach, the RepeatTests class rule can also be applied to individual test classes, for example:

@RunWith(FATRunner.class)
public class SomeTest extends FATServletClient {

    // Runs all tests in 'SomeTest' first without modification, and then again after upgrading
    // publish/servers/someServer/*.xml to use EE8 features
    @ClassRule
    public static RepeatTests r = RepeatTests.withoutModification()
                    .andWith(FeatureReplacementAction.EE8_FEATURES()
                             .forServers("someServer"));

You can have the repeated tests run in Full Fat only:

FeatureReplacementAction.EE8_FEATURES().fullFATOnly()

Or specify one of the FAT run modes (FULL, LITE, QUARANTINE, etc)

FeatureReplacementAction.EE8_FEATURES().withTestMode(TestMode.LITE)

Additionally, it is possible to exclude individual test methods or classes from a given repeat phase using the @SkipForRepeat annotation. For example:

    @Test
    @SkipForRepeat(SkipForRepeat.NO_MODIFICATION)
    public void testEE8Only() throws Exception {
        // This test will skip for the EE7 feature (i.e. NO_MODIFICATION) iteration
    }

    @Test
    @SkipForRepeat(SkipForRepeat.EE8_FEATURES)
    public void testEE7Only() throws Exception {
        // This test will skip for the EE8 feature iteration
    }

And finally, for maximum level of flexibility, it is possible to write custom actions by implementing the RepeatTestAction interface. For example, a FAT could be repeated with multiple database types in the following way:

public class RepeatTestsWithDB2 implements RepeatTestAction {

  public RepeatTestsWithDB2(/* DB2 database info */) { }

  @Override
  public boolean isEnabled() {  /* check if a DB2 database is available for use */ }

  @Override
  public void setup() { /* get and modify server.xml's to use DB2 database config */ }
}

And a custom action could be used as:

  @ClassRule 
  public static RepeatTests r = RepeatTests.withoutModification()
                                           .andWith(new RepeatTestsWithDB2(/* db2 database info */));

If helpful here is the componenttest.rules.repeater package source

Run a FAT test for a specific Repeat only

You can run a FAT test or bucket for a single repeat with the following command line flag:

-Dfat.test.repeat.only=<ID>

where is the FeatureReplacementAction’s ID.

example:

-Dfat.test.repeat.only=EE9_FEATURES

Repeating FAT tests for Jakarta EE 9

Jakarta EE 9 brings the "big bang" javax.* --> jakarta.* package namespace change. We have a repeat action called componenttest.rules.repeater.JakartaEE9Action in fattest.simplicity that will do 2 things:

  1. It will update all server.xml's in autoFVT/publish/ to use their Jakarta EE 9 equivalent feature (e.g. servlet-4.0 --> servlet-5.0)
  2. Any archives that are exported via ShrinkWrap will be automatically transformed to Jakarta EE packages (e.g. javax.servlet --> jakarta.servlet)

To enable the Jakarta EE 9 repeat action in a specific test class or entire FAT bucket, add the following items:

  1. Add a RepeatTests @ClassRule (if one is not present already) and append .andWith(new JakartaEE9Action()). If the RepeatTest rule is placed in the FATSuite.java it will repeat the entire bucket. If it is placed on a specific test class, it will repeat the tests in that class. 
    @ClassRule
public static RepeatTests r = RepeatTests.withoutModification() // run all tests as-is (e.g. EE8 features)
                    .andWith(new JakartaEE9Action()); // run all tests again with EE9 features+packages
  2. Add the following line in the FAT bucket's build.gradle file: 
    addRequiredLibraries.dependsOn addJakartaTransformer
  3. Update the bnd.bnd file of the FAT bucket to include the new EE 9 tested.features. For example: 
    tested.features: \
  servlet-5.0,\
  jsp-3.0
  4. (Optional) If certain tests don't apply to Jakarta EE 9, they may be skipped by adding the @SkipForRepeat(JakartaEE9Action.ID) annotation to the @Test method or class element.
  5. (Optional) If any extra files/archives need to be transformed that aren't server.xml's or archives created by ShrinkHelper, you can programmatically invoke the JakartaEE transformer like this: 
    if (JakartaEE9Action.isActive()) {
  Path someArchive = Paths.get(...);
  JakartaEE9Action.transformApp(someArchive);
}

Running FAT tests locally with different java levels

Normally everything in OpenLiberty runs/builds with JDK 8. If you want to build and run a FAT on a different java level locally, do the following:

run a different version of Java for a FAT.
Set up the java sdk on the PATH and JAVA_HOME then
./gradlew test.project:buildfat and then 
update bootstrapping.properties in **_test.project_**/build/lib/autoFVT directory 
(to point to the right Java)
reset your JAVA_HOME to be the right Java you want to test and 
run ant -f test.project/build/libs/autoFVT/TestBuild.xml

These instructions used to be:

export JAVA_HOME=/path/to/desired/jdk_home
export PATH=$JAVA_HOME/bin:$PATH
./gradlew --stop
./gradlew :com.ibm.ws.whatever_fat:buildandrun

NOTE: Stopping the gradle daemon (./gradlew --stop) is important, because it will force a new gradle daemon to be created next time you run a gradle command which will contain the PATH update.

Alternatively, if you only want the Liberty server to run with the controlled JDK (i.e. keep the build and the client side junit process on JDK 8), then instead do: (change the path to CD-Open wlp if running tests from there)

mkdir /path/to/open-liberty/dev/build.image/wlp/etc/
echo "JAVA_HOME=/path/to/desired/jdk_home" > /path/to/open-liberty/dev/build.image/wlp/etc/server.env

Skipping tests based on java level

Currently OpenLiberty supports Java 7 and 8, with all new features requiring a minimum of Java 8 to run. As a result, any FATs for new features must not run on Java 7.

There are 3 ways to skip FAT tests based on Java level:

  • Setting javac.source: 1.8 in the bnd.bnd file (i.e. compiling the FAT with Java 8) will indicate that the entire FAT should skip on Java 7.
  • Setting fat.minimum.java.level: 1.8 in the bnd.bnd file will indicate that the entire FAT should skip on Java 7.
  • Using the @MinimumJavaLevel(javaLevel = 8) annotation on a test class or method will cause the annotated element to skip on Java 7.

Error/Warning/FFDC scanning

Sometimes things can go wrong during a FAT test case that doesn't cause the JUnit test case to fail, such as an error or warning message in server logs. To catch this, whenever a LibertyServer is stopped, the messages.log file will be scanned for errors and warnings. If any errors or warnings are found, an exception is thrown which will trigger a class-level JUnit failure.

If a test generates errors or warnings that are expected to occur, then a list of expected errors can be passed into LibertyServer#stopServer. For example:

  @AfterClass
  public void tearDown() throws Exception {
    // Some test generated the *expected* error:  CWWK1234E: The server blew up because blah blah blah
    server.stopServer("CWWK1234E"); // Stop the server and indicate the 'CWWK1234E' error message was expected
  }

If a test generates an FFDC that is expected to occur, use the @ExpectedFFDC annotation. For example:

  @Test
  @ExpectedFFDC("java.lang.IllegalStateException")
  public void someErrorPathTest() {
    // Do something that causes a java.lang.IllegalStateException
  }

If a test generates an FFDC sometimes and we decide it's OK to allow, use the @AllowedFFDC annotation. Obviously, this annotation should be used sparingly, and @ExpectedFFDC should be used instead whenever possible. An example usage would be:

  @Test
  @AllowedFFDC // allows any FFDCs to occur
  public void someNastyErrorPathTest() {
    // Do something that could cause a variety of error paths to occur
  }
  
  @Test
  @AllowedFFDC("java.lang.IllegalStateException")
  public void someErrorPathTest() {
    // Do something that *sometimes* causes a java.lang.IllegalStateException
  }

NOTE: @AllowedFFDC() can be added at the servlet level to apply to all tests in that class, or at the test runner class to apply to all tests it runs.

Java 2 Security testing

OpenLiberty allows running with Java 2 Security (j2sec) enabled, and so the buildandrun task runs with j2sec enabled (-Dglobal.debug.java2.sec=true) by default to test this scenario.

This can be disabled by using -Dglobal.debug.java2.sec=false, for example:

./gradlew build.example_fat:buildandrun -Dglobal.debug.java2.sec=false

Even if there are only a few distinct j2sec issues, the log files can quickly become overwhelming due to the volume of logs per issue multiplied by the potential frequency of issues. It also can be a slow, iterative process to fix the issues one at a time.

To speed up this process, the SecurityManager installed from -Dglobal.debug.java2.sec=true will NOT blow up on j2sec issues, and will instead log a message.

After the FAT is done running, a report will be generated for each server instance in build/libs/autoFVT/ACEReport-<timestamp>.log, and even more detailed information about the full list of AccessControlException(s) will be contained within each of these server's console logs.

j2sec bootstrap.properties

The -D properties above cause the test framework to generate properties within each server's ${server.config.dir}/bootstrap.properties file, and so the properties in this file can be edited directly by the experienced user (as in the example below). Note because of the test framework writing to these properties, simply editing the bootstrap.properties in the test source will not necessarily help (since the framework will add its own values). See this Knowledge Center article for more info.

If a server should NOT run with java 2 security ever
Add this to the ${server.config.dir}/bootstrap.properties file for the server you want to permanently disable java 2 security for:

websphere.java.security.exempt=true

Using Docker containers with FATs via Testcontainers

Did you know? There is a sample testcontainers project that you can copy from when creating a new testcontainer dependent FAT test? build.example.testcontainers_fat

Sometimes a FAT test requires an external resource, such as a database. In the past, external resources have been manually maintained on external machines, but this can result in a number of issues:

  • The people who set up/maintained the external resource have moved on, and now nobody knows how to set up the resource
  • The external resource crashes, and needs to be manually restarted
  • Current tests and older service streams point to the same resources, but sometimes external resources need to be updated to newer versions as the FATs evolve

These issues are alleviated by putting external resources in Docker containers. Thanks to a library called Testcontainers, we can easily integrate our JUnit-based FATs with containers.

To use Testcontainers in a FAT

  1. Add the following runtime dependencies to your FAT's build.gradle file:

     addRequiredLibraries.dependsOn copyTestContainers

  2. Add the following buildtime dependencies to your FAT's bnd.bnd file -buildpath:

     -buildpath: io.openliberty.org.testcontainers;version=latest

  3. Next, define the containers you want to start using the Testcontainers API, for example:

    import org.testcontainers.containers.GenericContainer;
import org.testcontainers.containers.output.OutputFrame;
import com.ibm.websphere.simplicity.log.Log;
import componenttest.containers.SimpleLogConsumer;

// Can be added to the FATSuite to make the resource lifecycle bound to the entire
// FAT bucket. Or, you can add this to any JUnit test class and the container will
// be started just before the @BeforeClass and stopped after the @AfterClass 
@ClassRule
public static GenericContainer<?> myContainer = new GenericContainer<>("foo:1.0")
                                           .withEnv("SOME_ENV_VAR", "some value")
                                           .withLogConsumer(new SimpleLogConsumer(TestClass.class, "foo"));

By default, the FAT will run the containers on your local machine if you have Docker installed. If you do not have Docker installed, or if the FAT is running on a remote build engine, then an external pool of Docker host machines (IBM internal only) will be used. To force an external Docker host to be used, you can run the FAT with the -Dfat.test.use.remote.docker=true option.

For a simple example of using Testcontainers in a FAT, check out the ValidateCloudantTest from the com.ibm.ws.rest.handler.validator.cloudant_fat bucket. A more complex example can be found in the com.ibm.ws.jdbc_fat_postgresql or com.ibm.ws.cloudant_fat buckets.

For more details on using Testcontainers in general, check out testcontainers.org

Adding libraries to the FAT classpath

To add libraries to the FAT classpath - so that they are available from your test classes at runtime - add them under requiredLibs in your dependencies in your build.gradle file.

dependencies {
  requiredLibs 'javax.ws.rs:javax.ws.rs-api:2.0',
	'org.apache.cxf:cxf-core:3.1.16',
	'org.apache.cxf:cxf-rt-frontend-jaxrs:3.1.16',
	'org.apache.cxf:cxf-rt-rs-client:3.1.16',
	'org.apache.cxf:cxf-rt-rs-extension-providers:3.1.16',
	'org.apache.cxf:cxf-rt-transports-http:3.1.16'
}

Adding libraries to a specific location

Typically if your server needs a <library> of some sort, you you will need to put a binary dependency at some specific location. This can be accomplished with the following code in the build.gradle file.

configurations {
    // Defines a dependency group called "myLibs"
    // can have multiple dependency groups if needed
    myLibs
}

dependencies {
  // A dependency group can have multiple dependencies
  myLibs 'com.foo.somelib:1.0',
         'com.foo.anotherlib:2.0'
}

task copyMyLibs(type: Copy) {
  shouldRunAfter jar
  from configurations.myLibs
  into new File(autoFvtDir, 'publish/shared/resources/myLibs')
  // Optionally rename files as they are copied (useful for stripping off version numbers)
  rename 'somelib-.*.jar', 'somelib.jar'
}

// Hook into the standard 'addRequiredLibraries' task that runs at FAT build time
addRequiredLibraries.dependsOn copyMyLibs

Test Features

If your FAT is testing against a custom test feature, there is some project refactoring that will need to take place for your test feature to be available to the application during runtime.

  1. First create two new bnd files, {my_fat}.bnd and {my_feature}.bnd in the root of your project.
- com.ibm.ws.my_fat/
  ...
  - bnd.bnd             # Describes source folders and compile-time build dependencies
  - {my_fat}.bnd          # Describes the main fat bundle
  - {my_feature}.bnd      # Describes the user feature fat bundle
  1. In the bnd.bnd file enable sub-features by including the line -sub: *.bnd
  2. In the {my_fat}.bnd file add the following properties to differentiate it from the test-feature.
-include= ~../cnf/resources/bnd/bundle.props

bVersion=1.0

#This file defines the main fat bundle
Bundle-Name: {Project_Name}
Bundle-SymbolicName: {Project_Name}
  1. In the {my_feature}.bnd file add the following properties and configurations
-include= ~../cnf/resources/bnd/bundle.props
bVersion=1.0

#This file defines the {my_feature}-1.0 user feature bundle

Bundle-Name: {my_feature}
Bundle-SymbolicName: {feature.package}; singleton:=true
Bundle-Description: Bundles that tests a user feature that uses persistent executor; version=${bVersion}

Export-Package: {feature.package};version=1.0

Private-Package: {feature.package}.internal*

Include-Resource:\
 OSGI-INF/metatype/metatype.xml=test-bundles/userFeature/resources/OSGI-INF/metatype/metatype.xml

-dsannotations: \
  {feature.package}.{component},\
  {feature.package}.internal.{component}
  1. Add gradle dependency to build feature at runtime
dependencies {
  requiredLibs project(':com.ibm.ws.my_fat')
}

task copyFeatureBundle(type: Copy) {
  dependsOn jar
  from buildDir
  into new File(autoFvtDir, 'lib/LibertyFATTestFiles/bundles')
  include '{feature.package}.jar'
}

autoFVT {
  dependsOn copyFeatureBundle
}

Note: The point #5 above doesn't seem to be needed. To get the user feature definition to be installed during server startup, the key is to add the program logic to copy the feature bundle and the feature .mf to wlp/.../usr/extension/lib. For example, in the FAT test java file,

server.copyFileToLibertyInstallRoot("usr/extension/lib/", "bundles/{feature.package}.jar");
server.copyFileToLibertyInstallRoot("usr/extension/lib/features/", "features/{feature}.mf");

Once the sub bundles are built correctly, the LibertyFATTestFiles folder is generated under autoFVT/lib with its sub-folders of "bundles" and "features" containing the respective bundle jar and feature mf.

Clone this wiki locally