Device and software fragmentation are among the key factors to pay attention to during the development of a new software product. There’s no guarantee that a website or an app will run smoothly on all the existing devices by default. Even slight differences in hardware and software specifications can significantly affect the end-user experience. Thus, it is essential to run cross-device, cross-platform, and cross-browser checks. All these activities fit under the umbrella term compatibility testing services.
Theoretically, the more devices you use for manual software testing, the wider coverage you can achieve. Testing software on more devices should allow finding and eliminating a maximum possible number of defects. This idea sounds reasonable, but it doesn’t always work that way in practice. On the one hand, it is not possible to cover 100% of the market. On the other hand, it is not necessary. There are devices sold only in particular regions, and it is unlikely for your target audience to use them.
In other words, testing on different devices should involve an ultimate mix of smartphone and PC models, OS, and browser versions. And in most cases, it doesn’t mean all possible combinations. So how to cover enough scenarios without testing on too many devices and wasting time? Should a client or a QA company come up with this list? We’ll try to explain it all in this article.
How to Choose Devices, OS, and Browsers for Compatibility Testing?
The variety of devices and software they utilize keeps growing. So how to make sure you cover enough smartphones during mobile device testing to get accurate results? And what is the right way to test websites on different devices, when laptops and PCs come into the picture? Some might think the solution is to find a company with the largest catalog of real devices and test on everything.
To be honest, testing on everything is usually unreasonable for several reasons. Firstly, it will take a lot of time. Secondly, the QA process gets more expensive. Thirdly, and most importantly, it may be possible to gain the same insights into product quality utilizing fewer devices and software options. Testing on everything can be justified only in specific cases.
Who Should Choose Devices for Compatibility Testing?
Sometimes clients come to a QA company with a specific list of devices and other parameters. Thus, the parties only need to confirm the availability of those devices. If a client doesn’t have a specific request, it is a task for a software testing company to choose the combination that will work best.
In the first case, the list of devices is based on user data and the statistics the company has gathered. In the second case, a QA team chooses devices, OS, and browsers based on their experience and usage statistics.
Speaking from experience, over half of clients come without a precise list. A smaller share has particular requirements for the targeted devices, OS, and browsers. Finally, only some need to test on everything. Regardless of the request, we are always ready to prepare a strategy that will suit the product.
Compatibility Testing for Existing Products
Always start with market research and analysis. It is easier to determine device-OS-browser combinations for testing an existing product since you can tell from what platforms and browsers your app or website gets traffic. To learn this, look into Google Analytics. Get traffic overview with total visits and traffic by countries and sources in SimilarWeb. Use other data resources that will help to know your user and the tech stack they use.
Real Device Testing for New Products
If you don’t know your users yet, learn what’s popular on the market. Based on this data, you can choose five to ten devices with different software options. Start the research with the following questions:
- What are the leading device manufactures?
- What are the most common screen sizes?
- What are the most used OS and their versions?
- What are the most widely-used browsers?
- How does this data vary across different regions?
There are resources that provide the essential information for free. For instance, on StatCounter, you can find information on the device vendor, OS, and browser market share, screen resolution stats, and more. Users can view the information by region, country, time, etc. There are many insights that will be useful for website, web app, iOS, and Android application testing.
Some companies, like Perforce, occasionally share reports on mobile and web coverage index. They are centered around client data gathered by a company and illustrate general trends in particular regions.
More information in the following resources:
- Android Dashboards – an overview of characteristics of active devices.
- Apple’s Official Support Pages – data on iOS and iPad OS usage.
- ScientiaMobile – mobile overview report.
- InsightPortal – information about mobile and desktop OS versions, top devices, and desktop browsers used globally, in Europe, and North America.
Dealing With Less Popular Devices and Specifications
After learning the information above, you may wonder what to do with the rest of the browser and OS versions, smartphone and PC models that are used less frequently. In general, it is safe to assume they are capable and functional when you release a product. It also makes sense to test the basic functionality on recent browser and OS versions since not all users enable automatic updates. It goes for both website and mobile app testing.
A Risk-Based Approach In Real Device Testing
If you are aware of a certain problematic area, testing it is worth the time in most cases. It can be a particular browser or its version, OS version, screen resolution, or device model that is prone to defects.
When dealing with problematic areas, QA engineers usually rely on their previous experience. For example, Internet Explorer is known as a problematic browser. Though not widely used and soon to be discontinued, it is installed as a default option in Windows. While people are likely to use other browsers in daily life, it might make sense to test office software on Explorer.
Similarly, UI-related aspects are likely to break on devices with lower resolutions. You might want to test responsive mobile design on devices with low screen resolutions, like iPhone SE with its 640×1136-pixel display.
Compatibility Maths: How Many Devices Are Enough?
According to BrowserStack, there were 63,000 possible browser-platform-device combinations as of 2019. Along with sharing this data, they suggested a way to deal with fragmentation efficiently. As it turns out, you can reach 70%-80% of global device coverage by testing a website across:
- three device types (mobile, desktop, and tablet);
- two iOS versions;
- five stock Android versions and eight vendor implementations;
- six browser families;
- latest browser versions of each family;
- and ten screen resolutions.
That’s when you run website testing on different devices. If we are talking about mobile application testing, the list will be shorter. Keep in mind that you can combine the parameters above in different ways. In other words, there is no need to test on each possible combination. Just make sure to cover each option at least once.
Compatibility Under Tight Deadlines
Lack of time is not a reason to give up compatibility testing. It is essential to check at least several options before the release. For example, if your product is a website or a web application, check it on the latest versions of:
- Google Chrome, Firefox, and Microsoft Edge on Windows;
- Safari and Google Chrome on Mac;
- an iPhone and an Android phone;
- an iPad and an Android tablet.
If it is a mobile app that is due to be released, test it on at least two iPhone and two Android models, an iPad, and an Android tablet.
Based on how much time it takes to test on one device, you can extend this list if it’s possible. A lot depends on the product under test. If it is a big informative website, the defects would be pretty much identical regardless of device specs and software. Therefore, a mobile app testing service doesn’t need to cover a large number of combinations.
On What Devices, OS, and Browsers to Test?
We looked up some data on StatCounter to provide a better understanding of what is happening on the market (as for June 2021).
Leading Device Vendors
Popular Operating Systems
Widely-Used Browsers
To Sum Up
It is significant to design new software products with fragmentation in mind. Compatibility goes beyond iOS or Android device testing. It should also consider different browsers, operating systems, their versions, and various combinations. It will help you to gain confidence in the product and deliver the user experience your audience expects.
The good news is that efficient compatibility testing doesn’t require a large number of devices. It is the right combinations of devices, OS, and browsers chosen specifically for a software product that allows you to achieve vast coverage.