Mobile App Testing Strategy: Ensure Mobile App Success in 7 Steps

June 22, 2017 Sun Sneed

As agile development has become the norm—90% of organizations practice agile—mobile app testing and a clear mobile testing strategy become essential.

Agile product development with rapid release cycles allows companies to make their mobile products dynamic. The product becomes an experience in and of itself, establishing a feedback cycle with the end user instead of being static.

Unfortunately, agile has presented somewhat of a paradox for mobile product developers—while agile dev makes for rapid iteration, continuously refining the experience based on robust data gathered from user behavior and customer feedback, that agility sometimes comes at the expense of quality, specifically in two areas: user experience and data integrity.

User Experience

A high-quality user experience is the product of a combination of beautiful design, personalization and seamless, intuitive user flows. While agile developers are able to quickly build and change mobile app features to accommodate the evolving needs and wants of their users, this agility can often result in tunnel-vision when it comes to mobile app testing.

What can end up happening is, in an effort to quickly vet the functionality of a new feature before moving on to the next scheduled release, developers fail to take into account the effect new features will have on pre-existing features of the product.

This passing-over of the mobile app testing phase usually isn’t done out of ignorance, but rather out of duress: short release cycles make it difficult to set aside sufficient time to properly test each and every use case that could potentially break your mobile app.

The ultimate result is developers finding themselves facing serious challenges further down the line as unaddressed issues rapidly accrue interest on complexity debt with each additional release.

Data Integrity

Analytics data is both the end result and the catalyst of mobile app development. Data-driven DevOps teams collect and surface data in order to understand user behavior and how to optimize app performance.

Data becomes the lifeblood of DevOps strategy, and thus accurate data is a key objective.

With mobile SDKs making it relatively easy to implement analytics on your mobile app, mobile developers may fall into the trap of thinking they’re golden, with no need to validate the functionality of their analytics SDK through mobile app testing.

Unfortunately, if not properly validated, developers and product teams may fail to realize that not every screen is tagged, or that the analytics tool is not collecting the correct data—or collecting the data correctly.

Unlike a broken screen flow, errors in analytics data may not be as visibly obvious. You may not notice something is amiss until your data starts looking awry, but by then you’ve already lost valuable data that you have no way of recovering.

Plus, if you’re not keeping tabs on how your SDKs are affecting mobile app performance, it could result in a negative user experience.

Be Agile and Test Your App

If you’re not careful, agile development without a proper mobile testing strategy could result in exactly the problems you were hoping to avoid when going agile: excessive development complexity, negative user experience and long time-to-market.

Still, to remain agile you don’t have time to manually replicate thousands of quality assurance tests on each new iteration of your mobile app. Consequently, you need to trust in automated mobile app testing tools where you can continuously build a portfolio of use cases to test, all of which can be automatically run on each new iteration of your mobile app.

7 Steps to a Sound Mobile Testing Strategy

Strategy should always come before action. Building a mobile testing strategy during the early phases of development will ensure your app stays clean and effective for users.

Consider the following steps when developing your mobile testing strategy:

  1. Address device and OS fragmentation
  2. Manage size and “chatty-ness” of your app
  3. Test early, test often
  4. Employ mobile app testing along the process
  5. Use mobile analytics
  6. Test your analytics implementation
  7. Strategy—Start with the end in mind

1. Address Device and OS Fragmentation

The mobile market is divided across devices and platforms, and so is mobile app development. The two principal platforms dominate the market—99.6 percent of new smartphones run either Android or iOS.

Because both platforms hold a large share of the mobile market, most mobile-friendly organizations will build two separate apps, often created by two different teams. If proper analytics strategy isn’t developed and applied, disparate teams may create disparate definitions of conversion events and use different naming conventions.

To prevent disparate data, decide on unique user identifiers, conversion events and naming conventions for analytics variables that are independent of operating system. Share code as much as possible using IDEs like Xamarin (think hybrid app instead of two separate native apps).

2. Manage the size and “chatty-ness” of your app

Negative user experiences due to overly bulky mobile apps, or apps that spin up the network radio frequently, will result in rapid uninstalls and negative reviews. 50% of mobile users uninstall apps because they take up too much storage.

While mobile SDKs can extend the features of your app and are relatively easy to implement, too many is just too many. Be sure to monitor network traffic, look for ways to batch requests, and use long polling, SSE or websockets to minimize data transfer to keep your app performant and lean.

3. Test early, test often

Similar to the “release early, release often” mantra of agile developers, a “test early, test often” mentality will ensure you resolve issues earlier rather than later (which is must less costly). The sooner you identify problems through mobile app testing, the sooner you can resolve them.

Early development requires you to build a minimum viable product (MVP), and this will be key to ensuring your first-party code and third-party SDKs are working in tandem without any issues. Using a combination of code spot-checking and automated testing will allow you to identify points of failure during the initial development phases.

4. Employ mobile app testing along the process

There are myriad steps that go into developing, testing and iterating a successful mobile application. Integrating testing into each step of the process is agility insurance, protecting you against complexity debt.

Unit testing

Unit testing is the process of testing individual components of your code base, independent of other components. This allows you to isolate issues as you are adding new features to your technology.

Alpha/Beta testing

Testing the usability of your mobile app with real end users will help ensure you’re not only resolving technical issues, but also meeting the needs of your target user base.

Internal feedback can be very helpful, but if you can organize a customer advisory board to perform alpha testing on an early prototype, you’ll get much more relevant insight on your minimum viable product.

When you’ve resolved key concerns surfaced during alpha testing, you’ll be able to move on to beta testing, where users can expose unique use cases, usability issues and potential points of failure.

5. Use mobile analytics

Analytics is an integral part of any mobile testing strategy in that the data collected by analytics tools can help you identify bottlenecks, bugs and use cases that cause crashes. There are different categories of mobile analytics to consider, including:

  1. In-app Analytics
  2. A/B Testing
  3. Crash reporting
  4. App store analytics
  5. Voice of Customer analytics
  6. Ratings
  7. Push analytics

As a special note, consider the roles of real-time data vs. historical data:

Real-time data immediately transmits data based on behavioral events defined in your analytics variables, and allows you to respond immediately. Real-time data is important for location- and event-specific marketing, allowing you to reach a customer at the opportune moment.

There are some implications to be considered with real-time data, however. You will need to be prepared to process data rapidly. You will need to have some sort of automation or AI set up to respond to real-time data in order to get the true value from this data.

Historical data, on the other hand, is less time-sensitive, thus allowing you to focus less on immediate action and more on deeper analysis.

The way in which you intend to consume the data will determine which style of implementation you will need.

6. Test your mobile analytics implementation

Just like any other component of your mobile application, your analytics implementation can break. And if it breaks, you lose key data surrounding user behavior for as long as you remain unaware of the failure, plus the amount of time it takes you to identify what has broken and why.

The later you identify the issue, the higher the cost of remedying the challenge. Finding an issue in production costs 100x more than identifying the problem during the development phase.

(Source: isixsigma)

Spot-checking your app code to ensure everything is implemented correctly is one method to ensure your analytics code is working properly, but this sort of quality assurance is not scalable and is only a temporary solution.

7. Mobile Testing Strategy—Start with the end in mind

Involving the right people with the correct documentation from the very beginning will ensure you accomplish your strategy. It is best practice to establish a data governance council that is comprised of all data stakeholders. This council is headed by a chief data steward, who is responsible for ensuring the council meets frequently (monthly or quarterly) and that all analytics documentation is up to date.

An analytics strategy that goes undocumented is not much of a strategy. Having your KPIs, metrics and goals written out, along with clear and dynamic variable strategy documentation (sometimes called a solution design reference), will ensure your company has a true north for your testing compass to navigate towards.

As stated before, mobile app testing is agility insurance. If you want both your app and your development processes to remain dynamic, minimizing the number of bugs that slip through to a production environment will greatly increase the probability of success in an incredibly competitive mobile app market.

Introducing automation into your mobile testing strategy can make all the difference for an overwhelmed development team—or a team hoping to speed up release cycles by isolating broken code without having to manually comb through thousands of lines of code.

AppAssurance™ is ObservePoint’s patented technology for mobile app testing, capturing data SDK functionality, and validating your implementation against pre-defined compliance rules. Request a free sample audit of your app to see how automation can streamline testing processes.


About the Author

Sun Sneed

Sun is currently the Senior Product Manager at ObservePoint. She is passionate about internet products and marketing. Sun conceptualizes and drives change in an impactful and sustainable way. Sun currently leads product innovation for AppAssurance, ObservePoint’s mobile app tag and data quality platform. In past roles, Sun has contributed to the product innovation of Deutsche Telekom, T-Online International AG, and Fast Multimedia AG.

LinkedIn More Content by Sun Sneed
Previous Article
Using ObservePoint’s Rules Engine for Robust Tag Validation
Using ObservePoint’s Rules Engine for Robust Tag Validation

This article discusses how to use ObservePoint's powerful rules engine to customize tag validation across y...

Next Article
Identity Resolution Is Key to Digital Transformation
Identity Resolution Is Key to Digital Transformation

This article discusses the role of identity resolution in making your marketing strategy more contextual an...

Get a free 14-day trial with ObservePoint

Start Your Trial