Mobile test automation, an overview

I’m writing this article to give you an overview of some of the most commonly used mobile test automation tools available for iOS and Android, this is not an exhaustive list of tools available and i’m only going to cover the top 5 solutions (Counting the iOS and Android ‘default’ tools as one) I will also throw in my thoughts at the end. The ‘top 5’ was determined by their support in online test automation providers such as Appthwack, Saucelabs and Testdroid. you don’t need an online provider to use any of these solutions as they can all be run locally, but if you want to test against as many devices as possible the online services sure come in handy.

So if you’re looking into mobile test automation, hopefully this post can point you towards the right tool for your project.

Hit ‘Read more’ for the full overview.

Online test providers:
Appthwack
Saucelabs
Testdroid

Calabash (iOS and Android)
Calabash is one of the more well documented frameworks and is also now owned and maintained by Xamarin so has the backing of a (fairly sizeable) corporate entity while still remaining open source and completely free.

Setup will see you embedding their library into your application the benefit of this is that testers will not need access to the source of the application in order to run automated tests against it, but this does mean making a ‘special’ build of your application to get testing. Calabash is a multi-platform solution covering both iOS and Android, however in iOS your automated test area cannot exceed the scope of your application due to apple’s ‘Sandboxing’ approach, so for example; if you wanted to open up the iOS settings app and change a setting that could affect your applications functionality (such as disabling location services for example) this would have to be done manually. Tests run using cucumber (but can also be used with any ruby based framework.) and are written in ‘Gherkin’ a business readable domain specific language (Great for BDD).

Calabash links:
Calabash
Cucumber
Xamarin
Gherkin

KIF (iOS Only)
KIF stands for ‘Keep it Functional’, it makes use of undocumented apple APIs and iOS’s accessibility attributes to simulate user input, KIF tests are written in Objective-­C which makes it ideal for direct code based integration, one drawback of this is that although it is technically possible to run KIF tests without the source code, to make the most of KIF you really want access to the app’s source. KIF in my opinion is the easiest framework to setup for iOS as it can be installed through COCOAPODS and runs in Xcode as an XCTest target. KIF is an open source project (available through Github) and completely free, KIF is an iOS only solution.

KIF links:
KIF

Appium (iOS and Android)
Part of the Appium’s philosophy is that you should not need to add a third party library or perform a ‘special’ build of your application in order to test it, this is one of the primary plus points of appium as it is capable of testing the very same build that could become your ‘release’ build. Appium is both free and open source, it uses selenium and so is capable of running tests in any language that has a Selenium client library. Appium can run tests on both iOS and Android but is arguably not the most straightforward to setup, it is however a highly flexible framework and for this reason this solution seems to be most popular right now.

Appium links:
Appium
Seleium

Monkeytalk (iOS and Android)
Monkeytalk from ‘Cloud Monkey’ comes in two flavours, the first is their open ­source ‘Community edition’ and the second is their ‘Professional’ version. Both let you perform an unlimited amount of automated tests on your app. Unlike other test frameworks Monkeytalk comes with its own testing IDE used to run and report on tests. To setup tests for the community edition you need to add the MonkeyTalk ‘Agent’ to your project, once built testers will not need access to your application’s source. It’s also worth mentioning that using the Professional version of monkey talk you can inject the ‘Agent’ at runtime, meaning you don’t even need to make a special build of your application and your testers can start testing your app straight away, other ‘pro’ features include device emulator and extended test reporting. Both the Community and Professional versions of monkey talk cover iOS and Android. Pricing for the Professional version looks to be available on request so don’t expect it to be cheap!

Monkeytalk links:
Monkeytalk

Apple UI Automation (iOS Only)
Apple’s UI Automation is the proprietary one that comes bundled with Xcode. Though functional, i’ve found it to be slightly ‘fiddly’ to use with every UI component needing a fully qualified path to interact with it, that being said, Apple are improving the ease of use with every new release of Xcode. Test scripts are written in javascript and executed in the Xcode profiling automation tool, one issue that even apple’s very own solution does not resolve is the iOS issue of testing around ‘sandboxing’, tests written with UI Automation still hook onto the app process and sadly tests cannot exceed the scope of the application.

UI Automation links:
Apple documentation

Android UIAutomator (Android Only)
The scope of the Android UIAutomator is the entire operating system, this you the freedom of being able to write a full end-­to-­end test of any application on a real android device, although the scope of this framework is large writing tests can take longer than some of the other frameworks due to not always having a consistent UI attribute to hook onto. Test are written in Java and you are required to use the “UI Automator viewer” to find the UI attributes needed for the test. This is a free and open source solution and is included in the Android SDK. If you take a look at the article “UI Automator in eclipse” referenced below, you will see that tests are relatively straightforward to setup.

UI Automator links:
Google documentation
A good setup article “UI Automator in eclipse”

My Thoughts…
Right now there are some limitations to just how far automated testing can go in iOS, due to ‘sandboxing’ applications it can be hard to perform a full end-­to-­end test and also limits any testing to fairly basic functionality, also if you are developing anything other than a mobile application such as a library for third parties then automated testing is severely limited by sandboxing. Android is a completely different story with automated test frameworks available that give us full freedom within the OS.

When choosing a framework your selection is likely to come down to your working environment. If testers are not going to have access to the source code, for iOS I would recommend appium because it requires no third party libraries and you can test your true ‘release’ build, if the testing is for developers or someone working with the project source I would recommend KIF as its integration to a project and objective­-c test language makes it quick and easy for iOS devs to write tests.

For android my recommendation would be UIAutomator or possibly Appium, with UIAutomator giving you easy OS wide access. I have found UIAutomator slightly more convenient than Appium simply because in Appium you have to do some workarounds to achieve functionality that is very straightforward when using UIAutomator, one example is this, programmatically pressing the home button in UIAutomator is as simple as calling this method “getUiDevice().pressHome()” while Appium appears to have no direct method for this instead you would call something similar to the following “driver.execute_script(“mobile: keyevent”, {“keycode”: 82})” here you would need to find out the exact code for the hardware you would like to execute.

Luke Hines

Luke is a husband, father and software engineer who enjoys spending his time in a multitude of technologies. If he's not writing software then he's probably on a motorcycle.

Leave a Reply

Your email address will not be published. Required fields are marked *