Beta Testing: Everything you need to setup a Group of Testers for your App
First and foremost, let’s set the scene by a quick overview of Software Testing in relation to Mobile and Web Software Development, leading to the final and arguably most important stage: Beta Testing.
Functional Vs Non-Functional Testing
There are many forms of testing that will take place over the course of a software development cycle. Generally speaking, however, they can be split into two main categories:
- Functional — These are the different methodologies that test the functionality of the app specifically.
- Non-Functional — Here the developers will check non-functional aspects of your app like performance and reliability for example.
Beta testing rests within the Functional category. Thus this is what we will concentrate on for the remainder of the article.
The Main Functional Testing Types
I will be taking this from the perspective of a client who is employing a software developer or a development agency. Therefore I will ignore some of the more technical methods such as Code Testing and Backend Testing.
This is more of an umbrella term I personally use to explain to clients all the different types of testing that the developers perform on an app prior to the app being deployed onto a testing environment. FYI This is not really an industry standard term.
This is the process of ensuring that the app works as expected. This is achieved by testing that the various functions are inline with the business requirements. Its quite important that your developer(s) make the app available for you to test across key milestones. That way you can ensure that core functionality is inline with expectations.
At the end of the development period, you will enter the User Acceptance Testing (UAT) stage, where you will check the full flow of the app from start to finish. This could be regarded as Alpha or Beta Testing.
This is very similar to Beta Testing in that you are performing tests, predominantly at the end of the development phase with the goal of highlighting any bugs or defects to be fixed prior to launching. The main difference is that it’s performed by a much smaller group of trusted (usually internal) individuals. You have to be very careful with your Alpha Group because there will be issues, which you will not want to be publicised.
It is common practise to change minor aspects of your app’s user interface or functionality (UI/UX) at this stage.
Beta Testing is performed after the majority of bugs have been fixed so you can ensure that the app is stable. Even though you will highlight that this is not a “finished product”, you are essentially putting the app out into the public domain. If you do this too soon before you have dealt with any code issues, you could end up getting bad press due to a poor initial experience.
It is also regarded as the final stage of Acceptance Testing before releasing the app for public consumption. It usually takes place through the app store beta programs or in a monitored web environment. From a data point of view, this means it is very likely to be performed on a “live” production environment. However in certain situations you may setup a specialist staging environment that mimicks the Live environment but on a smaller scale.
There are many resources for Beta Testing out there that go into all the finer details so I will instead move onto how to setup the perfect Beta Testing Group.
Setting up a Beta Testing Group
I should note at this stage that there are actually two types of Beta Testing: Open & Closed. As you can guess, Open is available to anyone (you can access Beta Testing for apps straight from Google Play, for example). However Closed is invite only. From here on in, I will be assuming that we are setting up a Closed Beta Group.
Who should be your Beta Testers?
First and foremost you need to find your testers. Its really important that you target the right profile of person. Essentially you are looking to get your “Innovators”. These are those users that are inline with your customer profile, something you will have highlighted during the idea validation stage. The main difference between them and your larger user base is that they want to be the first to use and play with your app. This usually makes them tech savvy and forward thinking.
Resources For Beta Testing: The Guide
The Beta Testing process is really just an array of different resources that you will provide to your testers. It should be structured in such a way that it will guide them through the process.
Below I have outlined the resources I think are necessities for building a successful Beta Testing Group:
- A welcome email — outlines what the project is, what they are testing, policies and key links (signup questionnaire, FAQs, Facebook Group and so on).
- A signup questionnaire — you want to collect this info prior giving testers access to your app. For mobile app testing, you will probably need: general contact info, email linked to Apple or Google account on the testers phone, Phone make, model and current OS, plus any demographic data you want to collect.
- Guide to getting started — best to do a short video where you go through the motions and talk at the same time (one for iOS and one for Android). Its also useful to have a PDF guide. Send the relevant guides in an email after you receive answers from the signup questionnaire.
- An online FAQs list.
- Testing completion survey — after all the testing has concluded, this is where you are going to get the majority of your high end feedback.
- Reward for the testers — optional but suggested
- A platform for Testers to log issues — emails are a VERY BAD IDEA. I like to use Trello but that requires you to teach your testers another platform. Alternatives could be a web based Contact Form (this can be a fantastic tool because you highlight what fields are necessary), Google Spreadsheet, WhatsApp Group or Facebook group.
- A Medium to Communicate Company/Testing Updates — if you’re using a WhatsApp or Facebook Group then this would be perfect.
Getting Feedback from your Testers
As I mentioned in #7 above, your Beta Testers will need a platform to communicate any issues they are having. This could be to do with the Beta Testing process or the app itself.
Its therefore massively important that you set expectations, including:
- What they are testing — it sounds obviously but it’s really important to give testers clarity. Plus you might give different groups a different core function to test.
- What type of feedback you want — the good, the bad and the ugly?
- Where you want them to leave it.
- The format — for bugs best to have a screen video capture but at minimum you will need screenshot.
- What you will do with the feedback — its nice to illustrate the value of what they are doing.
The other thing you may want to consider is a reporting software to be installed onto the app itself like https://instabug.com/.
Managing the Process
Remember that your Beta Testers are essentially volunteers doing you a favour so you really have to live by the mantra of “the customer is always right”. Therefore you need to make the process of managing feedback as smooth as possible.
In particular, I would recommend:
- Having ready made online FAQs. These will make you life so much easier because you can generally direct users there to find their own solutions or to individual FAQs when they to solve specific known problems.
- Replying to their individual bits of feedback. This is critical to keeping them motivated.
- Keeping a line of communication up throughout. Its great of they can feel as though they are part of something “bigger”, possibly by giving them some inside info.
- Fix any critical issues straight away and release a new version. This shows that what they are doing is being actioned.
Finally after you get the tester to log their issue, you need to take that record and ensure that you have everything you need to investigate.
This means that you need to regularly review the feedback and request any further info where needed. Remember to say thanks to testers for the feedback when you don’t require any further info.
Next you need to place the issues into a log that can be reviewed by your developers. If you choose to give the testers access to the log (like Google Sheet or Trello), its a matter of moving it from a tester tab/list to a developer list. If you receive the issue on a group or via email, you will need to manually add the log into a convenient portal.
A Google Spreadsheet is great because it solves the issue of version control by allowing multiple people to edit the same file.
Beta Testing Consultants
Assuming that your developers cannot manage this process for you and you don’t have the time to do it yourself, I’d highly recommend hiring a consultant.
At my company, LaunchPadD, we offer services to our clients to aid them in this process. This could include everything from advice and setting up the group, all the way to finding the Beta Testers and running the actual program.
Regardless of whether you hire someone or do it yourself, make sure the process is well thought out. Beta testers are your Early Adopters so if you impress them more will follow.