We are available for hire, find out more.

RSS Feed

Writing scenarios for Guerrilla usability testing

By John ~ Wednesday, 29 May 2013

I’ll be honest, I enjoy the experience of running usability testing sessions the most if I get to test existing designs that I haven’t been involved in creating but that’s not always possible. Thanks to books like Rocket Surgery Made Easy, though, I am a confident moderator and have over the years manged to detach myself to some extent by managing expectations right at the start of the project and also by taking a few deep breaths beforehand and repeating the mantra, “The designer is dead”.

However there’s no getting away from the fact that I’m usually too emotionally attached to be truly objective and I will often recommend independent testing if possible. Fortunatley I have been in the privileged position of being able to hire indepenedent moderators to do usability testing and on one occasion, whilst working with the exceptional Canonical web team on developer.ubuntu.com, our external moderator Joe Knowles handed me a real gem that I use all the time now.

Whilst he and I were working out how to go about extending the reach of the testing by adding a bit of remote testing, Joe suggested we get the participants to talk about a project they are planning or about to start, or something they’d like to do in the near future.

You can read Joe’s full notes at the bottom of this post but for now these are the main points he outlined for me about why it’s better to let a user do something real:

  • Users are far more inclined to behave naturally if they are doing something they normally would
  • Users aren’t always that good at pretending, or doing things in a hypothetical way
  • Assumptions made by designers won’t always be correct
  • Assumptions may miss important bits - parts or whole scenarios
  • Real users with real scenarios give the research credibility
  • Credibility gives greater buy-in in the business
  • It’s more compelling

Joe explains that this does create a bit of extra work before hand but the pay off when you are doing this stuff yourself is worth it.

Most recently I did a round of testing for a holiday marketplace website and just to add to the fun we did the recruiting ourselves too (this is difficult and I recommend getting a recruitment specialist to do this for you if you can) and as part of the screener we asked if people had any holiday plans for this year or next and also what their favourite holiday has been. Recruiting this way being what it is, unreliable, we didn’t get absolutely ideal candidates for each session so we had to adapt.

We did get one perfect fit however and the resulting session (which we did at the participants house, streaming the session via Skype) was compelling as this genuinely potential customer tried to do something very real on the site. If there was tension it was everyone watching hoping this user would actually purchase the holiday there and then.

It’s worth noting that as Joe’s notes below suggest, we let the participant go on a ‘natural journey’ which did prove tricky to facilitate as at one point we ended up testing the email thread on the fly and not the website itself. We did have to stop the session eventually and move the participant over to the website and continue from there.

For another participant we based the session around what the participant did for a living (personal trainer) and a scenario involving arranging a fitness holiday for a dream client.

For another we asked him to tell us about an ideal holiday and wrote a scenario based around that and asked him go ahead and start planning it.

In one case a participant got quite frustrated and just gave up. At this point we tried to negate some of his frustration by very verbally abandoning the scenario and confessing we’d lost him as a customer. We then apologised to the participant and asked if he’d mind just trying to achieve a few small things on the site for the remainder of the session, which seemed to have the desired effect of removing some of the tension.

In all cases what I felt happened was the user was more engaged with the test because it meant something, even if it wasn’t quite as real as we’d have liked.

These sessions really can be tricky to run and you always need a back up plan for if it goes wrong (and your wits about you to change things on the fly) but I find again and again that this approach to Guerilla usability testing extremely valuable and doubly so when you are faced with trying to recruit participants too.

Joe’s full notes

Writing scenarios based on prior knowledge (no matter how extensive) will always require the author to make certain assumptions. Empathy is important to begin the process in the first place, but it is more than likely that the designer writing scenarios will often miss important aspects of how users do things, if not whole scenarios altogether. Best practice is to get users to do things as they normally would, at least initially. The problem with this is that they may then take off and go through a journey which you’re not looking to test. So what you might do is find out the sort of thing they usually do (if they have to do it in the near future then even better), and base the scenario around that. For example, you may have in mind certain things they have to do - on a holiday website this might be that they have to look for a flight and a five star hotel - and let them choose the destination and anything else that might be important, such as whether the hotel provides free Wi-Fi. This will ensure they have bought into the task and have some genuine interest in what they’re looking for, while ensuring you get them doing the things you need to test.

The other thing you might do is get them to do is take a natural journey, just watching them and picking up what is important to them through speaking aloud. Then get them to undertake a number of scenarios. You’ll see after the first few whether your idea of what they want to achieve, and what they actually want to do is similar in any way. The danger is that without finding out what it is they want to do, you could end up testing irrelevant scenarios, and finding solutions to problems that users don’t really care about.

Alternatively, if you’ve not got time to do natural use plus scenarios pre-written, it would be best to have discussions at the beginning of the session, finding out verbally what it is they like to do, or want to do, before tailoring the scenarios you’ve already written on the fly. This will be quicker than them taking you through a natural journey, but they may miss out, or forget, certain things that are important.

Testing users with realistic scenarios also gives the research a lot more credibility; with credibility the work that is ultimately done is more likely to get buy-in within the company involved, and less likely to be dismissed as the disconnected whim of a designer. Getting users to undertake tasks and scenarios that are realistic is also much more compelling. You can tell when watching a 30 second video clip how bought-in to the task they’re doing is. Credibility and validity are destroyed if your user completes a task and follows it up with “but I’d never actually do that sort of thing”.