Skip to content

A tricky case with Application Tests

Jump to bottom
AliSoftware edited this page Nov 18, 2014 · 15 revisions

TL;DR: read only the "Summary of the two recommended configurations" paragraph.

  • Two very different kind of tests
    • Non-Hosted Tests
    • Hosted Tests
  • Use the right type of tests to fit your needs
  • Tricky case when using Hosted (Application) Tests
  • Summary of the two recommended configurations
    • If you don't need UI testing
    • If you do need UI testing
  • A workaround if you really need it

Two very different kind of tests

In Xcode you can perform two kind of tests when you have a Test Bundle : Unit Tests (independent) and Application Tests (hosted).

You select if your Tests will be independent or hosted by your app by selecting your Test Target in Xcode and choosing the target/host of the test bundle in the dropdown menu under the "General" tab.

Non-Hosted Tests (Unit Tests)

When you select "None" in the drop-down menu, your Test Bundle will be launched on its own. Your application won't be launched at all and there will only be your SomeFooAppTests bundle loaded in memory.

That's the classic case and generally the one you need.

In such case, there is nothing special to care about, the OHHTTPStubs class will only be loaded once, there will be only one set of stubs, you setup your OHHTTPStubs generally in your -(void)setUp method or inside each -(void)testXXX method and you are good to go, everything works like expected.

When using this kind of tests, you may even add OHHTTPStubs to your Application target if you need to stub requests when running the application for real (for example if some WebServices are not available yet and you need to stub them until the WS is is production), those stubs won't impact the ones you use when running your tests. The stubs you would then setup in your application:didFinishLaunchingWithOptions: (or some place else) will only be setup and called when you run you app, and the ones you setup in your test bundle (in -(void)setUp or some other place) will only be configured when you run your tests.

In this configuration, you will have to add the classes you want to test (like your model classes for example) to your Test Target (in addition to having them in your App Target) so that they got compiled with the Test Target and that you can use and test them.

Hosted Tests (Application Tests)

When you select your application name in the drop-down menu, here is what Xcode does when you ask it to run tests:

  • Xcode launches the application in the Simulator
  • The application:didFinishLaunchingWithOptions: method of your application gets called as usual when the app launch
  • Xcode then loads the Test bundle, like a "plugin" (in fact it uses the OSX Plugin Architecture described here) into the app binary — as a "dynamically loaded bundle" hosted by the app — then executes the Unit Tests inside the application.

These tests are generally only suited for UI testing, not for logic testing.

At that point, both the App bundle and the Test bundle are loaded in memory. And in fact, if you linked OHHTTPStubs (and any other library) with both the App target and the Test target, the library will be loaded twice and there will be two instances loaded concurrently in memory, one by the App bundle and one by the Test bundle loaded as a "plugin" (as illustrated by the figure above)!

That's why it gets tricky…

(Spoiler: the solution is to avoid linking OHHTTPStubs to the test target so that it is only loaded once)

Note that this is how Xcode Hosted Test Bundle works, and the following is true for any library you happen to link with both bundles/targets, not just OHHTTPStubs.

In this configuration, you won't have to add the classes you want to test (like your model classes for example) to your Test Target because they will already be compiled and loaded by your App Bundle that hosts your Test Bundle.

Use the right type of tests to fit your needs

"Hosted Tests" (or Application Tests) is generally only suitable for when you need to do some UI testing. If you only want to perform functional/logic tests, then go back to standard Unit Tests by selecting "None" from the dropdown menu in the "General" tab of your test bundle.

Tricky case when using Hosted (Application) Tests

The main tricks to be aware of when using Hosted Tests are that:

  • If you linked OHHTTPStubs with both the app and the test bundle, there will be two independent instances of OHHTTPStubs loaded at runtime.
  • Every code loaded by the Application Bundle (like code from your model classes) will use the instance of OHHTTPStubs loaded by the application
  • Every code loaded by the Test Bundle (like code in your test suite) will use the instance of OHHTTPStubs loaded by the test bundle

That may or may not be what you want.

If you want to use some stubs in your application code in order to stub a WebService that does not exist yet (e.g. because the team working on the WebServices are not ready yet), and that stubs needs to be in place whether you run the application for real or you run your tests, then that might be a way to go. You may add additional stubs in your Test Bundle, call [OHHTTPStubs removeAllStubs] from your test bundle in -(void)tearDown and all, that will remove stubs managed by the test bundle (as you run it from the OHHTTPStubs instance loaded by the test bundle) but won't affect the stubs managed by the app bundle (which uses a different instance of OHHTTPStubs).

If you want to have different stubs for when you run your application code and when you run your tests, and thus if you don't want the stubs used by your Application bundle to be in place when you test your app, then you need a different strategy:

  • Maybe Hosted tests is then not what you are looking for, and maybe switching back to "None" in the dropdown of the "General" tab of your Test target is what you need. If you don't need to test the UI but only the app logic (model classes, logic and algorithms but not app UI) then Hosted tests are probably not for you.
  • If you still need Hosted (Application) tests because you will need to test some UI, you may setup the stubs you want to use when loading your app in some other place than applicationDidFinishLaunching, like some other place that only gets called when the app is running for real and not when it is launched to host UI tests

But the most relevant thing to do is probably to remove the OHHTTPStubs from the libraries linked against your Test Bundle, because the library would already have been loaded by the app bundle itself!

Summary of the two recommended configurations

If you don't need UI testing

  • Select "None" in the dropdown menu in the "General" tab of your test target. The app won't be launched to run your test and only test bundle will be loaded.
  • Add the classes you need to test (like model classes etc) to your test target (in addition to your app target of course) to be able to access and test them
  • Link OHHTTPStubs to your test target to be able to use it during your tests
  • You may also link OHHTTPStubs to your app target if you need to stub some requests when your app is running, like to mock some WebService that is not available yet

If you do need UI testing

  • Select your app name in the dropdown menu in the "General" tab of your test target. That way, the app will be launched and will then load the test bundle as a "plugin" and host it to run the tests
  • Don't add the classes you need to test (like model classes etc) to your test target, as they will already be loaded by your app bundle that will host the tests during testing
  • If you already linked OHHTTPStubs to your app target, don't link it to your test target too, as it will already be loaded by your app bundle that will host the test bundle during testing.
  • If you don't need to link OHHTTPStubs to your app target because you don't need to mock any WebService when running the app for real but only need to stub requests during tests, link OHHTTPStubs to your test target but not to your app target.
  • In either case, as the test bundle will be hosted/loaded by the app bundle, don't link OHHTTPStubs (as any other library) in both targets, as it would load it twice at runtime and you would end up with two distinct instances

A workaround if you really need it

Again, the solution that generally best fits when you really need Hosted Tests (that gets the app launched before starting the tests) is to not link the OHHTTPStubs library to the test target but only to the app target, as the test bundle will be hosted by the app bundle at runtime anyway.

That is actually true for every other library: if it is already linked against your app bundle, you shouldn't also link it to your test bundle provided that it will be hosted by the app bundle itself.

But if you really need two distinct instances OHHTTPStubs loaded by both the app and the test bundle, and really want to link OHHTTPStubs to both targets to have these two independent instances, then be aware that you may access the instance loaded by the App bundle from the Test bundle, using some tricks like this:

Class AppOHHTTPStubs = [[NSBundle mainBundle] classNamed:@"OHHTTPStubs"];

Using this in the context of the code loaded by the test bundle, this AppOHHTTPStubs will give you access to the OHHTTPStubs class instance loaded by the app bundle, whereas the OHHTTPStubs symbol will still point to the class instance loaded by the test bundle. Therefore, you could call [AppOHHTTPStubs removeAllStubs] to remove all stubs that could have been setup by the code of the app like from application:didFinishLaunchingWithOptions:, without impacting stubs setup in your own test bundle.

See also related issue #47 where all this is discussed, especially this comment that explains what happens when using Hosted/Application tests

Clone this wiki locally