5 Keys To Continuous Unattended Testing and Mobile Test Automation That Work

 In the world of continuous unattended testing, a developer's claim "Well it works on my device" is not acceptable. Mobile is by far more fragmented than Desktop/Web platforms due to the variety of device types, OS versions, and the environment on which it operates (network impact, roaming, battery stretch etc.).

When you test mobile, it is not sufficient to test on one or even a few devices, but instead test apps on several devices that adequately cover your end-users' device pool. How can you keep up with testing apps on so many devices, you ask? Easy. Build a working mobile test automation strategy that runs the same script across all of your devices, making your Dev/QA teams confident that they are delivering the best app to end-users.

From the experience Perfecto gathers everyday, we came up with the must-haves for building a continuous, unattended mobile test automation strategy that provides immediate feedback to its Dev/QA teams.

1. Relevant Device Selection

It is one of the toughest dilemmas in today's market to identify the right device X OS mix on which teams should develop and test their mobile app against. When building your prioritized list of mobile devices, consider the right criteria, including:

1. Target market and competition/End-Users - On which devices and OS versions your end-users actually use the most (based on organizational analytic data)
2. Generic market statistics - Top sold devices (smartphones and Tablets) in the relevant regions. Such data can be retrieved from analyst research such as IDC, Flurry, ComScore and others.
3. Change rate and market dynamics - As we all see, the mobile market is in constant change and your teams need to "listen" to the trends and changes- adopt them and change the strategy accordingly

2. Continuous Unattended Testing Brings Agile Development and Faster Feedback

1. Mandates mobile OS system level control - If the script cannot initiate events outside of the AUT (App under test) it cannot really test the app as the end-user will use it
2. Incoming alerts and error handling are also mandating system level control out side of the AUT
3. Hybrid objects support - automation framework should support both OS level objects (DOM for web and hybrid apps) and Visual objects for full use case support
4. Device logs, rich media reports (Video and Visual) and vitals (Memory status, Battery etc.) are critical evidence to Dev/DevOps teams as feedback to act upon

3. Device and Browser Elasticity

As your mobile app project grows/shrinks based on your SDLC phase, your test automation solution needs to be flexible enough to accommodate necessary changes and provide the needed support to your team (even more critical in cases of distributed teams).

4. Continuous Integration is Key for Cross Team Collaboration

Whether your Agile/Dev/QA teams are using ALM tools (HP, Microsoft, etc.) or OSS (Selenium, Robotium, JUnit) tools, your automation framework solution should be able to connect to these tools and bridge any gaps in the daily continuous work flow. Use what your team knows - this is the best way to release mobile apps faster without compromising quality.

5. Testing On Real Mobile Devices and Browsers

It is imperative to understand that testing on something other than real devices in your critical automation scripts and as your mobile app evolves is a huge risk. Only real devices running formal OEM OS flavors against real networks reveal an app's true functionality in the same way your end-user will experience it.

To conclude, see below (Fig 1), a check list of mobile testing pillars which need to be part of your Mobile App Testing Plan.