How reliable are your usability guidelines?

in Usability testing

Benchmarking products or websites is a task that usability professionals are often called upon to do. Expert reviews using guidelines is a common technique.

One problem with guidelines is that different experts can interpret them in different ways. For example, “Have a clear link to the home page” is a commonly quoted guideline, but what is “clear” to one usability expert may be unclear to another. I might pass a website on this guideline, but you might fail it. Then what?

This inconsistency risks clients complaining that your guidelines are simply subjective.

Showing that your guidelines can be consistently applied requires different experts to apply your guidelines the same way. In statistical terms, you want strong inter-rater reliability. You can measure the consistency between experts by calculating a Cohen’s kappa statistic.

If you want strong guidelines then you should hire a usability expert to improve the reliability of your guidelines.

Previous post:

Next post: