Henrik Rexed (LinkedIn) is a Performance Specialist at Neotys. He has been orchestrating and conducting performance tests for over 10 years, delivering projects in all contexts including extremely large Cloud testing on the most demanding business areas such as trading applications, video on demand (adaptive streaming), sports websites, etc.
According to a survey conducted by WebDAM, there are over 224 million monthly active mobile app users in the United States. With so many mobile applications available, consumers are searching for the best and fastest performing apps, which is why testing your mobile application is extremely important. But mobile performance testing can be quite expensive if you take into account your team’s need to invest in functional testing tools, performance testing tools, devices and more.
But what is the best way to test your mobile application properly for performance? In this post I will discuss the use of emulators and real devices for mobile application performance testing. Both of these tools have equal importance in mobile application testing, it all depends on what type of results you would like to see.
Testing Performance On Mobile Applications Using Emulators – As the name suggests, mobile emulators are software platforms, installed on a laptop or desktop, that imitate a mobile platform or OS environment from the point of view of the application running within the OS. In other words, apps think they are running on a real phone, not a PC. Although emulators are considered to be close to the real environment, they cannot entirely replace real devices. For example, if you are using an emulator to test an application that utilizes the phone’s accelerometer, it is highly likely the application might not work on the real device as expected. There also might be a discrepancy in the performance of your application running on the same OS across different devices – something which emulators are not able to capture.
Emulators facilitate the testing and development process to a great extent. They are useful for validating functional behaviors that are not specific to device or carrier. They also provide the ability to simulate situations that are difficult to recreate on a device. For example, if you want to test how your application performs on a device with low battery it can easily be imitated on the emulators. And of course using emulators saves money, most of them are actually free, and the requirement of acquiring multiple real devices – as well as the services plans behind them – is greatly reduced. This is especially impactful in early-stage testing.
Testing Mobile Applications Using Real Devices – Real devices, unlike emulators, allow the tester to physically switch from device to device in order to test the performance of the application. Testing with a device allows you test your app with live networks. However, you cannot test the signal strength, latency, bandwidth and packet loss of multiple devices connected to the network, that has to be done with a type of network emulation tool. That being said, testing your application on a network is crucial as most apps depend on the mobile network for its operation and performance.
The biggest drawback of using real devices is the cost. If you are simply testing the performance of the application a couple devices might suffice. However, one major factor real devices fail to address is the load placed on applications at any given moment. In order to test the load on an app using real devices you would need thousands of devices all set up in different locations accessing the application at the same time, which isn’t feasible in most cases.
Load Testing Your Mobile Application – One factor testers tend to forget is that when you record a scenario through a device or emulator, you are working with a playback scenario. You’ve recorded the traffic directed towards the app, which helps you understand the request the device is handling. But this doesn’t actually give you an accurate picture for how much load the application can actually handle.
Which leads me to discuss the fact that one of the most effective uses of an emulator tool is pairing the tool with a load testing tool. With your mobile application you can expect users are accessing your app from different geographical regions, using different networks, all expecting the same level of performance. Which is something devices can’t imitate accurately, especially since you need an ample amount of devices at your disposal to mimic the load which is simply not cost efficient. The only way you will know if your app performs well under load is to test it.
When you use a load testing tool with device simulation capabilities it will do two things: simulate browsers and browser capabilities. When a browser sends a request for resource to a web server, it identifies itself via the user-agent header sent with each request. This header contains information about the browser and the platform on which it is running. Servers use this information to deliver different versions of the content based on the client system.
Many web applications deliver different content to mobile users and desktop users. Some further differentiate mobile users into subgroups based on information in the user-agent header, delivering less text and smaller images to devices with small screens. This can lead to bandwidth consumption and loading times that vary widely with the browser and platform being used.
As a result, the ability to manipulate the user-agent header is essential not only for recording test scenarios, but also for playing them back. Tools that lack this capability will fail to retrieve the appropriate content from the server.
Always Pay Attention To End User Experience – It’s important to remember how your application is being received by the end user. End user experience should be considered when testing mobile applications. For example, if you come to discover your application quickly drains your consumers’ device battery, ultimately your user will become annoyed and find another application, which you do not want to happen.
Henrik Rexed (LinkedIn) is a Performance Specialist at Neotys. He has been orchestrating and conducting performance tests for over 10 years, delivering projects in all contexts including extremely large Cloud testing on the most demanding business areas such as trading applications, video on demand (adaptive streaming), sports websites, etc. Prior to Neotys, Henrik worked as .NET architect for Logica and Performance testing expert on large accounts in a variety of industries including insurance, automotive, retail and energy. Amongst the numerous initiatives he worked on, Henrik has built the performance Center of Excellence (CoE) handling load testing of all the European branches of a major French insurance company.
Since 2005, Neotys has been helping its clients in more than 60 countries to ensure their applications’ reliability, performance and quality. NeoLoad, load testing solution, increases productivity allowing you to perform your tests faster, while providing pertinent analyses of test results and full support for all new technologies. With a growing customer base in the United States, and over 1,000 customers in more than 60 countries, NeoLoad is fast becoming the industry’s web load testing tool of choice. Renowned customers in North America, EMEA and Asia rely on NeoLoad to improve the performance and quality of their web applications