Mobile Testing Brain Dump

A friend asked me for advice on testing, and I thought my response might be helpful for the rest of the world. This is simply a brain dump, with very basic structure, and there might be some minor inaccuracies. I encourage corrections, and will update the post with them as needed. :)

Each platform has it's own testing hell. First unit testing:

For Android, unit testing is a real pain. By default, Google recommends that you run your unit tests on an emulator or real device, even if they don't require any UI components. This is because there are no Android libraries that run directly on the development box. Thus, you're best bet is to segment your Java code into as much pure Java as possible and write pure-Java unit tests for that. Then, write Android-specific unit tests for the code that uses Android libraries. You can write those with JUnit, and they'll run on an emulator or device. If this isn't an option, you can look into Robolectric, which is an attempt to create a fake Android environment that runs on the JVM. I recommend against Robolectric because it doesn't have the exact same behavior as a real device, and doesn't support the latest Android (or it didn't back in 2014 anyway).

For iOS unit testing, you basically must use XCTest. There are other frameworks like Kiwi or Specta, but they're just BDD frameworks that wrap XCTest by artificially creating test objects based on their own method and assertion specification patterns. I recommend using plain XCTest because Apple hasn't had great support in the past for running unit tests. Thus, if you stray far from Apple's recommended path, you're likely to find that stuff doesn't work between releases of Xcode. On my last project, we used ordinary XCTest and things worked fine. You can easily add OCHamcrest, which Jon Reid maintains (he's a great guy and very helpful if you have questions). OCHamcrest is just a better way of performing assertions, but it doesn't actually alter the testing structure, so I think it'll still remain pretty compatible with each new version of Xcode. If you need to mock, I recommend OCMock.

For UI interaction testing:

For Android UI interaction testing, you can use essentially 2 different options: Espresso and UIAutomator. You use espresso if you want to run tests exclusively within your own app. You use UIAutomator if you want to run tests that might call out to many different apps (for example, if you have a twitter client, you might write a UIAutomator app that opens the Photo Gallery, presses the Share button, and shares a photo to your twitter client). With espresso, you have access to objects within your application because the tests run within your application's process. With UIAutomator, the tests run outside your application's process, so you must use only the UI itself to perform actions and verify results. UIAutomator uses Android's accessibility framework to navigate the DOM and extract values from UI controls.

For iOS UI interaction testing, things are unfortunately a bit complicated. Up until iOS8, Apple recommended the UIAutomation tool in Instruments. UIAutomation seems like it works for simple tasks, but it starts breaking down by not recognizing views on the screen or not properly tapping the right things. Worse, you must write UIAutomation tests with javascript. Also, UIAutomation doesn't support interacting with other apps or with system alerts, so you can't grant permission for your app to use the GPS, Photos, etc, or interact with other apps. Finally, there is no simple support for running UIAutomation tests from the command line, and there is no actual unit testing framework for it, though primitives exist that you can use to build your own.

Because of these problems with UIAutomation, Square created KIF. It runs your tests within XCTest (which itself just runs within an instance of your app). It relies on some non-public API for interacting with the UI. It worked well up to iOS8, but I've never run it against iOS9, so I don't know if it works there. It doesn't support interacting with system alerts or with other apps because your tests are confined to your process.

Starting with iOS9, Apple created XCUIAutomation, which allows you to write UIAutomation-style tests within an XCTest-like context. I've never used it, so I don't know how well it works, but I've talked to people at Apple who said that despite XCUIAutomation's similar name and APIs, a new team worked on it and completely rewrote it, so maybe it works better. I don't know if it will support interacting with iOS system alerts or other apps, but I doubt it.

In short, if you want to run on iOS7, iOS8, and iOS9, KIF might be your best option. If you want the best official support, XCUIAutomation might be your best option.

Finally, if you're doing UI interaction testing, you should try to run all your tests on as many different OS/Hardware combinations as possible. Amazon and Google have options for this, but Google's is in private beta, and only supports Android. Thus, I recommend trying Amazon's. I haven't used it yet, but since I'm an Amazon employee (by way of Twitch), I plan to give it a try as soon as I get some time to work on UI Interaction testing.

However, even though all these tools for UI interaction testing exist, I recommend not relying too heavily on it. I've tried on many occasions with many different systems to do UI interaction testing (including on Swing, Eclipse, and Selenium), and I've found that UI interaction testing always tends to be brittle, regardless of the framework or GUI toolkit. Generally, front-ends change a lot, and when they change, they tend to break all the underlying tests. You can mitigate that a bit by writing an abstraction over your UI to perform basic tasks (sort of like Cucumber  encourages you to do with Rails apps), but also adds to the cost of testing. You can theoretically get a lot of benefit from UI interaction testing, but I've never seen it work well on a really fine-grained level.