toggle menu Close Menu
Testing/QA

Setting up Your Mobile Testing Lab

Part 1: Determining and Sourcing your Devices

You have spent months building the perfect website. The layout is pixel perfect, the site loads fast and the content is relevant and fresh. But is your site perfect? How much time are you devoting to testing before you launch? If you are not fortunate enough to work in an area where you have access to an open device lab then purchasing actual devices for testing might be the only option.

According to Pew Research Center , “64% of American adults now own a smartphone of some kind, up from 35% in the spring of 2011. Smartphone ownership is especially high among younger Americans, as well as those with relatively high income and education levels.” (Smith, 2015). With the rise in smartphone ownership and use, mobile device testing is even more important than ever.

Google is also changing the way that it ranks mobile sites. As of April 21, 2015 mobile-friendly sites will be given search priority over standard non-mobile pages within mobile search results. Designing and testing for mobile devices is becoming even more critical than traditional desktop testing.

Mobile Phone

Actual Device Vs. Simulator

There are several reasons why you might be inclined to use a simulator rather than an actual device, the most important being cost of the devices. Given that a new unlocked device could cost upwards of $500, using a simulator seems like a good idea. However, testing on a simulator can lead to erroneous results and may show bugs/issues that are not an issue on an actual device. For example, we had an issue where validation was triggering on the credit card form on the simulator, but on an actual iPhone it was working correctly. We typically only use a simulator for devices that are not as popular or for a brand new device. For example, we use a simulator for iPhone 6s since that is a brand new device and an unlocked version costs over $650.

Picking your Devices

It is important to give some consideration to which devices you are designing for, and will be testing on, BEFORE you begin your project. You might want to start with your client’s expectations and requirements, but be careful when asking your client what devices they would like to use for testing. You may get an answer like “We want it to work everywhere”. That’s great as long as you have a dedicated army of testing minions and a truckload of money to buy all of those devices. A better approach might be to start with a base list of devices that both you and your client can agree upon and then expand outward as necessary.

Benjamins

We typically test on a minimum of four smartphones and three tablets for each project. However, that number may expanded based on the client's wishes or the scope of the project. We base our device selection on the latest user trends and sales and we to look at the quarterly Google Analytics mobile stats to review which devices are being used specifically on our client’s website and select our devices accordingly.

Even though it seems like a new device is released every week we tend to only purchase 2-3 phones a year. According to Forbes magazine, 51% of iPhone owners will upgrade their phone as soon as their provider allows it? (usually every two years), while 47% of users will wait until their phone is obsolete before upgrading. 40% of Android users would exchange their phone every two years while 58% would wait until their phone stops working.

We are currently looking to purchase the iPhone 6 and the new 6s was released a few months ago. Also keep in mind that unless the hardware is vastly different a phone that is a few months old is more than adequate for most testing scenarios.

Below is our current list of in-house devices:

  • iPhone 5
  • iPhone 4s
  • Samsung Galaxy 5
  • Samsung S3
  • Samsung S2
  • HTC One
  • iPad 3
  • iPad 2
  • iPad mini
  • Kindle Fire HD
  • Microsoft Surface Pro
  • Nexus 6

Where to find your devices

Now that you have identified which devices you want to purchase, how do you find devices without spending thousands of dollars? Thanks to the Internet and sites like Gazelle and Swappa, you should be able to pick up a gently used device for a reasonable sum. You might also want to check out your local cell phone stores. As people upgrade their devices to the latest and greatest device they have overstock of older devices. You might also want to look around your office. I bet someone is about to upgrade their device. Offer to buy your employees’ outdated devices. Not only can you get a newer device, but you also have someone on staff who can show you how to use the device if you are not familiar with it.

In my next post I’ll look at ways to organize and store all your testing devices.

Kristin Compton
Production & QA
or drop us a note and say hello!