Nearly every week I am asked, “why is mobile testing so hard?” or “how can we make mobile testing easier?”. Testing for mobile is a difficult challenge and while the latter question is ongoing and something that should always be challenged, the former often is answered with something nebulous like, “well, there are a lot of devices” or something to that effect – which, to me, is insufficient and neglects a higher level of quality. I try to never form an answer without sufficient data to support my claims, yet often I find myself searching for a solid piece of evidence that will illuminate why it is so difficult to test applications or websites on mobile devices.
Over the years, as I search for evidence, I have come across many articles that have tried to bring clarity to the question of mobile testing, how best to approach it, the pitfalls and how best to craft a clear and executable strategy. Nearly all of them have fallen short in one way or another, but this past week, I discovered an article from the folks over at Open Signal, where they at least try to get their arms around one of the primary contributing factors to mobile testing complexity: Android Fragmentation.
For me, Android has always been an interesting platform. It offers much more flexibility or freedom (if you like) as a developer to build applications openly and easily. In contrast, Apple’s closed garden may seem like limiting to some developers, but after reading Open Signal’s article, as well as seeing their supporting charts, it is clear that the extent to which Android’s flexibility exists to serve an open and free market, it also increases overall complexity within the device and operating system manufacturers.
As a developer, this variation in device/operating system pairing is a core component to what causes mobile testing to be either incredibly difficult or incredibly costly to ensure. Anecdotally, the evidence was always there and it is echoed when someone answers, “well, there are a lot of devices” to “why is mobile testing so hard”, but until I was able to see the graphical representation of how much fragmentation exists within the Android platform, I hadn’t expected the scale of the problem to be as great as it is.
Happily, we know that not everyone uses every device or operating system pairing, but reading an article such as this helps us better understand, with data, the complexities involved within the problem and it helps ensure that we can communicate clearly with our clients and developers. Additionally, as there is likely not going to be a slowing in fragmentation, Open Signal’s article reinforces the need to always use analytics data to make sure that your application is targeted for users that want to communicate with you, rather than simply trying to address the entire spectrum of devices and operating systems. An “all or nothing” strategy will quickly dwindle your budget on testing, rather than creating new and innovative features that are worthy of your brand.
