Bitbar – Automated cloud-based testing for Android

November 15, 2012

One of the things we enjoy at SwiftKey is meeting other companies, large and small, and sharing knowledge with them.

Today we played host to Jouko Kaasila and Henri Kivelä from Bitbar, who came to talk to us about automated cloud-based testing for Android.

Michele Sama (second from right), SwiftKey’s Test Architect, will tell us a little about what they talked about.

Software testing is a critical activity for any software company. From a purely commercial perspective bugs and slow performance are the main reasons for which users stop using a piece of software. From a managerial perspective the cost, in terms of development time, of detecting and fixing bugs increases exponentially the longer a bug exists in the code.

The combination of continuous integration and automated testing is one of the most powerful instruments which quality engineers have to improve the quality of their software during any stage of development.

Developing for mobile platforms such Android proposes an additional challenge to traditional continuous integration systems: the variety of devices on which your software must work. It may sound obvious but even if your tests pass on an emulator your application may still have problems with certain devices. Such problems are known as “integration issues” and are detected by the so called “integration tests”.

Here at SwiftKey we support 7 versions of Android, on at least 10 different screen sizes with over 50 languages. To test all of those combinations we would have to run our integration tests on 7 * 10 * 50 = 3,500 configurations. This would be eventually possible by using, for instance, Jenkins and by running our tests on a matrix of configurations using the Android emulator.

Should we run our tests 3,500 times?

The answer is probably not. First, not all of those combinations exist, which means that in case of errors you will need to exclude false positives. Second, a configured emulator is still an emulator and it will not detect device-related integration faults. Third, when a single test fails then you will have 3,500 failures!

How should we run our integration tests then?

That is simple. You should run them on all the devices that you support. How much will it cost you? How difficult would it be to maintain them? Well… a lot.

The people at BitBar think they have the answer.

The idea behind Testdroid is simple. They own a cloud of devices, including more than 150 different models, and they let you run your tests on those devices using their web frontend. You can then configure a set of “must work” devices (including the most popular devices) and you run your tests on those at every commit using a Jenkins plugin.

The next step is to test the integration with more of the market devices to see which ones your application supports. Once again we can create two cluster of phones, one with all the “fully supported” devices and one with the “not fully supported” ones. You can then, daily or weekly or every now and then, check which devices you support and make sure that all the tests pass in those ones that are fully supported. On each test run Testdroid also collects CPU and memory usage of your execution, making it possible for quality engineers to check for performance issues or memory leaks. You can also select the language with which to configure your devices before the execution.

The latest addition to the tools available in Testdroid cloud is the Android app crawler. In a similar way in which a web crawler browses a web site, the app crawler explores your application view by view and generates a graph representation of each view, including a screenshot for each device. Being able to obtain screenshots from all the devices under test without manually clicking through is very useful.

One thing that scares me as a quality engineer is the adoption of new tools and library. Before adopting one I always need to write down what I call the “adoption risk analysis”. I absolutely do not want to be forced to ask the team to rewrite all the tests because we have to change a testing tool. With Testdroid this is absolutely not an issue. Testdroid runs (and generates) normal Android instrumentation tests which means that there are no adoption (and no leave) costs.

To thank Jouko and Henri for their visit, after all they came all the way from Finland to give us a talk, we decided to show them a preview of few new tools which we have developed internally.

Keith is a pre-build configuration tool which uses a set of Python scripts to change the Android resource files for a specific build. Keith lets you define products (targets) and modifiers (rules) and lets you apply modifiers to each product.

SlimChimp is a Java extension of MonkeyRunner inspired by ChimpChat which lets you install and control Android applications exploring the elements on the screen of the device. With SlimChimp not only we can install and enable SwiftKey as a keyboard during our tests but we can also test it on third party applications of which we do not have the source code.

Chauffeur is an Android testing support tool which avoids code duplication in your test setUp and tearDown method by configuring and restoring your test environment for you.

You will probably hear more about these tools in near future.

You can find out more about Bitbar and what they do at http://testdroid.com/.