A few weeks ago we shared a first article on how we build design at Alan:
“4 designer tricks to launch an online insurance in less than 9 months”.
One of the important tool described in the post is user testing. I wanted to go deeper with this practical guide on user testing for start ups!
Keep in mind this is the summary of my experience running tests at Orange, Withings and Alan. It’s certainly not perfect, but it will provide a solid base if you have no experience at all.
Please give us your feedback and comments!
So now: how to prepare, perform and analyse the results of a user test?
You are trying to learn how users use your product in real life. But, like in quantum physics, in user testing, the observation modifies the observed.
You need to reduce the bias linked to the test as much as possible.
First, make sure that you are in the right state of mind:
In order to do that, it is crucial to:
Note: If you’re evaluating an existing product, the best is to observe real users performing real tasks. At Alan, we sometimes go to client companies and just observe employees subscribing to Alan after receiving an invitation email.
Tests are time consuming, you don’t want to do too much of them, but to have meaningful feedback, you need a sample that is large enough.
So how many tests should you perform?
5 is the answer.
Why ? This is a practical guide; for theory, go here, or here.
Keep in mind that if you can’t do 5, 1 is always way better than 0.
The more tests you perform, the larger number of potential issues you can spot and the better you can assess the importance of each of them. 5 is just a sweet spot; extra tests have diminishing returns.
The actual test may take between a few minutes and an hour.
Don’t go beyond 60 minutes. The sweet spot is about 30 minutes of real action (that’s about as long as someone can really pay attention to something), add 15 minutes before and after for intro, questions etc…
For the whole test session, I usually plan for:
If you want the user test to be useful, plan for time to implement the feedback. At least a few days, and you may need another test session to make sure you fixed your problems.
If you need fresh users that don’t know your product yet:
If you want to test things that require users that already use your product, then you need to recruit among your user base.
This does not cover 100% of situations. Sometimes, you’ll need to be creative to find users. Keep in mind that when you launch your product, you’ll need to make contact with your users anyway, so take the time to figure it out!
Should you offer a reward? 💰
As much as possible, try not to. Rewards create a bias in users’ motivation.
If you offer a reward, it should not be too important, otherwise, you’ll end up with testers that are only here for the money. They are not representative of your target (usually) and won’t give you candid feedback.
The equivalent of 30 to 60€ (in gift cards for example) is a fair amount.
I do it in spreadsheet like this one:
Nothing fancy:
Once scheduled, send a proper calendar invitation to participants.
They are human (that’s the point, ain’t it?), they will forget the appointment, so set a reminder as well.
Prepare an email with all the informations for the test:
Include some warnings:
It’s better if the user doesn’t try to understand your product before the test.
So specifically ask them not to look for information, go to your website or try your product in advance.Best is when they ignore who the test is for or what it’s about.
If planned in advance, send a reminder at the beginning of the week to make sure they didn’t forget.
There are a lot of things you may want to test, but keep 2 things in mind:
For example, for our first test session at Alan we wanted to know, for our company product:
Taking those questions, we define a list of high level tasks the user will perform with our product:
Our intro narrative was the following:
You are not happy with the current health insurance of your company. You are planning to change. You are reading a TechCrunch article about a new Health Insurance. Here is the article.
At this point, we created a fake TechCrunch article about Alan that stated realistically what an article of the kind might say about us, with a link to our test landing page. Make sure you don’t give more information to the participants than the level of knowledge you expect an actual user to have at this point.
If the narrative is natural, you should not have to manage the articulation between the different high level tasks too much. Sometimes, you may need to write narrative for transitions.
In our case, once the participant explored the homepage, he would sometimes naturally go to the subscription process. Sometimes, we’d just guide him with a “Let’s say you are convinced and want to join Alan. What would you do?”.
The protocol will serve as a guide for you during the test.
Write on it:
"Thank you so much for being here and helping us out.
When we develop products, at some point we need some fresh views on what we do. That is why you are here.
A few recommendations before we start:
I will ask a lot of questions, but you are not being tested, the product is. I’ll just be trying to figure out how you see it.
As much as possible, speak out loud, anything that goes through your mind, hesitations, questions, your understanding of what is happening, how you feel. We know it ain’t natural, I’ll remind you during the test.
As much as possible, behave like you would normally. Don’t feel obliged to read everything if you usually don’t, don’t rush reading if you usually would take your time, we’ll wait.
The product won’t break. Do anything you’d do in a normal situation.
I won’t answer questions during the test. If there is a technical problem, I will intervene. We’ll answer all your questions at the end.
The test will last about 30 minutes".
We usually add (because it’s true for us):
Everything that happens in the test is fake. We won’t record your data, you won’t sign anything that is real.
As much as possible, use real informations when asked (real ID number, IBAN, etc…).
Take one person in your team or a friend, use her as a tester and perform the test. Make sure:
Congrats, you’re ready!
Now let’s do the test. 💫
As a way to limit the bias of your test (and because you are a nice person), you want to keep your user in a normal state of being. Being in a foreign environment, feeling tested or judged goes against that.
Should you record your test session?
Reasons behind it are multiple:
One huge benefit of the user testing is that it gets the debates inside product teams out of the hypothetical. Theories gets crushed, ideas trashed.
Some people have a hard time accepting it and may not trust the results of the tests. The solution is either:
There are two problems with recordings:
My advice:
The first team member drives the test, following the protocol
The second team member takes notes. 📝
On the note-taking side, you can use several tools, a document, a spreadsheet, a Trello board. Use the same nomenclature you used in the protocol. Watch for:
Create a test report file, go through your test notes and gather the feedback, step by step.
List the problems spotted, their recurrence (how many times was the problem spotted), and add a gravity level. For gravity, I use this grid:
Also write on the protocol the interesting things participants may have said at this point, and your understanding of their state of mind.
Obviously, problems that are blocking are important, problems that have a high recurrence as well, and both blocking and recurrent are your top priority.
Keep in mind that all problems spotted don’t need to be fixed.
A problem that happened only once, even if medium or blocking, may not need to be fixed. Note that the problem was spotted, try to find other ways to validate that it was just a punctual issue (watch your analytics for example).
If you had a big problem (blocking and recurrent), fix and run a new test session to make sure you solved the issue.
At the beginning of the test report, add a summary of your feelings about the test.
Highlight the good and bad stuff that happened, duplicate the top priority items on top.
For every problem that needs fixing, add an action to take (something to implement, a task for the design team, a copy change to make,…)
Make sure everyone in your team has access to the test report, the protocol, the screener and the raw notes. Let them know they can ask questions.
Thanks for reading this, I hope it’ll help you build a strong, user-proof product.
One other challenge we have not talked about in this post is how to organise user tests on a regular basis, how to recruit participants continuously and how to mix it into your product organisation.
We are still figuring this out, and we may talk about our conclusions in a later post!