Project teams need time to digest and reflect upon your usability testing results.
I recently ran a workshop with a project team in which I presented usability testing results. That presentation was the first half of the workshop. In the second half we worked on specific design changes. Both sessions went very smoothly and we had really positive outcomes from our efforts.
But one thing that struck me was the discussion that the project team needed after I’d presented the usability testing results in the first half of the workshop. They needed time and space as a team to form their own shared understanding of the results they’d just seen. They didn’t want to jump straight into hard-core design-thinking. They needed time to digest and reflect on the findings - to ponder what the key issues were and mull over tricky issues.
And they also needed time to check the health of the team, as if saying “Wow, some of that was tough, are we ok… yes, good, let’s move on then.”
So present your results, and then allow plenty of time for the project team to digest and discuss the findings together. Let them come up for air.
Instead of using the mean to report average task times from usability evaluations, you should use the median or geometric mean, according to new research by Jeff Sauro in Denver, USA.
The most common way to find the ‘average’ task time from a set of users is to calculate the mean. But average task times based on the mean are too high because there are usually one or two users who run into problems on a task and skew the results.
For example, if you have collected the following task times from five users:
100, 101, 102, 103 and 104 seconds, then the mean is 102.
But if a 6th user ran into problems and took 200 seconds then the average task time becomes 118. However, 118 seconds is not a true reflection of how users as a whole performed - all five of the original users were below this ‘average’!
It is generally safer to use the median. Here the median is 102.5 for all six users - a much better measure of how the users performed.
But Jeff Sauro’s research shows that you’ll get even more reliable averages if you use the geometric mean. In Excel it’s the geomean function. With the the six users above, the geometric mean is 114 but for real usability task times (as opposed to the trivial data above) Jeff has has shown that the geometric mean provides better estimates of the true average even than the median (and certainly better than the mean).
By choice I use Fieldworks near London Bridge for recruiting participants for user testing or research.
I’ve used them for years and they typically respond well to a brief and keep me well informed of how recruitment is progressing.
Other people have mentioned to me that they use:
Let me know how you get on with any of them.
Silver projects are important to your company, but not as mission-critical as Gold projects.
Silver projects may make up the bulk of your projects and include those which:
- Do not affect your customers directly
- Do not affect primary personas
- Affect secondary tasks or lesser-used functions
- Address low-level interface issues rather than structural issues that really impinge on the user experience
Examples of Silver projects: Staff intranet, advanced functions used by expert users, account maintenance pages, tidying-up the visual design of a few pages.
While these projects need serious usability support to get right, they are not as business-critical as Gold projects.
There’s two common approaches to supporting Silver projects:
- Put mid-weight or less-experienced staff on the project (saving your best practitioners for Gold projects).
- Provide the critical structure of the design, for example, sketched paper prototypes. Then allow other developers to design the detail and have your trained usability practitioners review it and give guidance. This puts usability staff in control of critical decisions without taking all their time and allows coaching of less experienced developers.
On low-priority Bronze projects (e.g. simple pages that impact few users, or where it’s difficult to go far wrong) you should provide an evaluative function. This means that non-usability specialists do the design work, while your team provides advice and guidance during design iterations. In this way, you can support an entire project with just a few days from one practitioner.
As the manager of an internal User Experience team you need to prioritise the projects to which you assign your staff. If you give a bit of help to all projects you’ll spread your staff too thinly - and the business-critical projects will suffer.
Prioritise your projects into one of three levels, Gold, Silver, or Bronze.
How to identify Gold projects
- Are mission critical to the business - they will make a big difference to your company’s on-going success
- Work on tasks or functionality needed by lots of users
- Fix big problems that really hit the bottom line: usability headaches that hit lots of users lots of times
- Stakes are high - improving success rates, satisfaction, or efficiency will bring in big money or mitigate severe risks.
Examples of Gold projects: Developing personas, prioritising functionality on the home page, product search and comparison tools, removing barriers in the check-out process.
Put you best staff on Gold projects. If you’re understaffed and quality and timelines are critical, get outside top-notch contracters in.
Next we’ll look at Silver projects - the important but not critical ones that make up the majority of your projects.
It’s funny, reflecting on what you learned on a project, what seemed to work, what felt right.
Yesterday I finished a five-week user research project in the employment and recruitment area, working alongside another researcher, Ash. I thought back how we’d tried hard to establish a rapport with the client at that first meeting, listened to his problems, his anxieties, his hopes.
I seem to go through the stages of user research almost without thinking, guided by years of experience I guess. Interview the client, collect all related documents, find out if anyone else in the organisation is working on a related area, speak with internal staff - all the initial wide lens work to gather all the thoughts, issues, and materials of potential relevance.
With a sharper focus, in the next stage of our research, our ‘real’ interview questions were chosen by hypothesising and proposing issues that could be explored. We interviewed 29 participants. The whole process is one of clarification, of sniffing-out data like blood hounds, of ensuring that the data gathered is grounded in people’s experience. And on it goes. Immersive and satisfying.
But why so satisfying? I believe it’s because we build a model of reality, one that has a coherent meaning to us, something we can believe in, something we can share with others.
More than anything, though, you get that uniquely satisfying ‘Aha!’ moment when all the parts, concepts, and relationships you’ve been working with finally form a Gestalt totality from what is already known.
One of the oldest usability principles tells you to “Speak the user’s language”.
Are you getting increasingly annoyed at our train companies’ idiot-speak?
In the past few weeks I’ve been to Scotland, Hull, and Sheffield. On each trip a poor “member of the cabin crew” parroted out some pre-scripted drivel. Please have your travel documents ready. My what? Please refrain from smoking in the vestibule-ettes. The what-ettes? Your next calling point is Boxford. The what-point? For your convenience today’s cabin crew are muppets who will be serving a vast array of beverages… and on and on it goes.
Their sterile idiot-speak creates a gulf of affinity between us and them. Words matter - they create meaning and meanings give rise to emotions. In this case negative ones.
We sit there in their hermetically-sealed coffins, not just disengaged from the train company, but disengaged from travelling and, worse, disengaged from life itself.
Oh dear. And relaaaax… 
Creating a feeling of connectedness to others is one of the elements of a positive user experience. Here I’m talking about creating a user experience in the true sense: stimulating positive mental states in the minds of your users.
A sense of belonging - how we feel and interact with those around us - is important to feeling safe and our well-being. We have evolved to attend and think about being with others who are like us. We seek out others with shared values, ideas and interests to feel this sense of connectedness.
Sharing and wanting to share is fundamental to the very nature of being human. This is one reason the internet is such a major development in human communications - because we are passionate sharers, seeking others like us.
The Handbook of Global User Research has just been written by companies within the UX Alliance, an association of user experience agencies, who provide clients with a global service.
From the table of contents and a sample chapter, the book seems comprehensive in scope, covering traditional research techniques as well as eye tracking, unmoderated testing, and web analytics. This is reassuring and will be very interesting. A couple of people have posted positive reviews on Amazon.com
If you do global research yourself, you might find a host of tips and lessons. If you commission global research, you’ll have a better idea of what to expect (and ask for) from your agency.
Buy if for £34.19 from Amazon in the UK.
Diary studies are a useful way for understanding customers’ needs, motivations, and behaviors.
While they are cheaper than observing users over a long period of time, paper diaries tend to have problems:
- Typically filled out once a day only (so subject to recall inaccuracy)
- Rich detail may be forgotten or edited out by customers
- Self-referential - talked about through a time of day the customer thinks is important
I’m investigating software for running online diary studies to overcome these problems. A simple blogging or chatting tool such as tumblr may work well for online diary studies. I would set up a blog for each of my customers and they will fill them out.
Advantages of online diaries appear to be:
- Customers may be more likely to record their on-going activities throughout the day with a web-based tool - especially for web-based research.
- On-going recording is likely to give a more accurate account of their day, i.e. less self-referential.
- Allows researchers to monitor customers’ contributions and encourage diary-keeping.
- Facilitates quicker data logging and aggregation
If you have any experiences of using something such as Tumblr (or Twitter) as a diary or cultural probe, I’d be interested to hear your experiences.