PM101 Chp List
Chapter 11 – How to conduct Usability Testing?
As a Product Manager, it is critical to understand how users would interact with the product — How easy or difficult it is for the user to use the product.
That’s what Usability Testing helps a team with. You can test your product with your target users in a controlled environment and identify vital changes to make your product successful.
What is Usability Testing?
A usability testing is a black box testing technique that consists of gathering unbiased participants representing your target users and asking them to perform specific tasks to test your hypothesis. The testing interface can range from a paper interface to screen mockups or an MVP.
It reveals whether the users feel comfortable with your app or website according to following parameters –
– Navigation & Layout
Usability testing tests the following usage components :
- Learnability: How easy it is to for first-time users to use the product (learn the product)?
- Efficiency: How fast can an experienced user accomplish the task?
- Memorability: When a user returns to the product after a period of time, does the user remember enough to use it effectively the next time?
- Errors: How many errors does the user make? The types & severity levels of the errors, and whether the user can recover from them.
- Satisfaction: How much does the user like using the product and the Possibility of the user recommending the product (NPS)?
How to conduct Usability Testing?
A. Develop Test Plan
A. Step 1 — Identify scope and reasons for testing.
Scope — What are you testing? Example, prototype, navigation, content, etc.
Purpose — What are the concerns and user goals? Example, “Can users navigate to the important information” or “Will users find the search box in current location?”
A. Step 2 — Create Test Scripts
Schedule, Duration & Location — When & Where you will do the test? Duration of the test?
Session & Equipment — Describe the session (including the length) and equipment requirements to facilitate the session.
Role — Include a list of the staff who will participate in the usability testing and what role each will play.
A. Step 3 — Recruit Users
Participants — Indicate the number and personas of the user on which you want to conduct the usability testing.
If you already have users — Identify people who are actively engaged with your product and send a short message/email.
If you have no users — Think about places where your target audience gather either in person or online. Reach out to special interest clubs, friends of friends, Meetup, Reddit, online forums, etc.
Here’s a sample message that you can use for user recruitment –
Hello . I’m the Product Manager of and I noticed that you are an avid user of . I would love to get your feedback on — it’s an app/site that helps with . Would you have 15 minutes to get on a call? If so, do let me know an appropriate time to call you.
A. Step 4 — Create Test Scenarios
Describe the Scenarios that will be used during the testing. The scenarios consist of the user stories and the context that outlines the tasks needed to accomplish the user goal.
The scenarios shouldn’t be generic and should test the hypothesis.
Let’s take a look at following scenarios –
Scenario 1 — Go to Booking.com and book a hotel
Scenario 2 — Go to Booking.com and book a budget hotel (< $200 / night) in Singapore (Near Marina Bay) for 25th — 30th Oct.
Scenario 3 — Go to Booking.com; Type Singapore in Location and select dates. Check hotels near Marina Bay on the map. Sort the hotel by Price and select a hotel (< $200 / night) and book a room.
Scenario 2 is a valid scenario as it is neither generic (as Scenario 1) and nor instructional (as Scenario 3).
A. Step 5 — Identify Test Metrics
- Effectiveness — % of users completing the task
- Task Time — Amount of time it takes for the participant to complete the task.
- Non Critical Errors — Errors from which the participants can recover and complete the task (less efficiently). For example, opening the wrong navigation menu item.
- Critical Errors — Errors from which the participants cannot recover and has to abandon the task. Example, unable to find Buy Now button.
- Error-Free Rate: % of participants who complete the task without any errors.
- Subjective Measures: These evaluations are self-reported participant ratings for satisfaction, ease of use, ease of finding information, etc where participants rate the measure on a 5 to 7-point Likert scale.
- Likes, Dislikes and Recommendations: Participants provide what they liked most about the site, what they liked least about the site, and recommendations for improving the site.
B. Conduct Usability Test
During the test, you should never lead the participant and should take detailed note during the session.
You could use Lookback.io as your test recording tool.
It allows teams to record onscreen activity within mobile apps and websites. It can also record the user’s face and voice.
C. Analyse the result
Go through the collected data. You could use the following format to arrange the data –
- Arrange the data in consumable format
- Identify the problems and categorize them in various themes.
- Brainstorm solutions around these problems and develop an action plan.
Usability Testing Techniques –
Heuristic Evaluation UX
What — A lightweight evaluation that allows you to identify usability issues without the involvement of users. It is based on Nielsen’s Ten Heuristics. Check — http://www.uxcheck.co/
The evaluation is generally done by UX Experts.
One to One interview session with the session. (Can be recorded or simulated)
Determine where the user clicks or how far they scroll and where are the user eyes focussed on a screen.
Compare two different versions to see which one performs better based on the goal
Want to learn more?
Enroll into Product Management:A-Z – The complete product management online course with 25+ chapters, 40+ live assignments and product interview tips. Built with inputs from experienced Product Managers working @ Top Tech firms like Google, Adobe, Amazon; the course has 500+ enrollments from 40+ countries.
Check out PM Interview Questions – An exhaustive list of unique Product Management interview questions answered by product experts.