PURE Usability Method | Jack D. Davis

PURE Usability Evaluation Method

PURE (Pragmatic Usability Rating by Experts) is a method to evaluate the usability of a workflow and identify opportunities for improvements. It's more lightweight than traditional usability testing, but still provides useful results. In this method, usability experts assign one or more quantitative ratings to a workflow based on a set of criteria and then combine all these ratings into a final score and easy-to-understand visual representation. For more details on the PURE method, please check out Nielsen Norman's page on the subject.

The slides below are excerpts from a report-out on a PURE study that I conducted while at The MathWorks.

User Goal

Before running a PURE study, you define a user for your usability experts to consider when they judge the workflow. You can use a persona or describe the characteristics of the user, and the goal they're trying to achieve.

In this study, the user is an aerospace engineer designing the fuselage of a new aircraft. They are testing the safety of the fuselage given a particular evasive maneuver--can the fuselage "hold together" given the gravitational and aerodynamic forces that will be applied?

Map the workflow

Next, map out and make visible the current workflow for your expert reviewers. There are three main tasks the user needs to do: (1) create the study (2) run simulations of the scenarios being tested (3) view the results to answer the design question. Each task has several sub-steps, as shown in the image.

Map the workflow

Task 2: run simulations

Map the workflow

Task 3: view results

Facilitate the session

During the actual PURE session, gather together the usability experts. Then, demonstrate the user's workflow, step-by-step, and the experts give each step a usability rating on a scale of 1-3 (1 being best, 3 being worst).

Typically, the experts are UX researchers, designers, or others in the company who are familiar with the product and end-users. Yes, this is intended as an internal method and not done with end-users. In this way, it differs from usability testing, and hence why it is "pragmatic"; it's faster, cheaper, and easier to conduct. Is usability testing better? Probably. But it's also a lot more costly. And your experts, although internal, can provide valuable insights.

Rating Scale

The rating scale used to rate the usability of each step of the demonstrated workflow.

Results

This is the PURE scorecard, showing the actual ratings that the experts gave each step in the workflow.

In the bottom area, each box represents a step in the workflow, and the number in each box is the usability rating given by the experts (during the session, there's a technique to bring the multiple experts to agree on one number per step).

In the upper left is the total score for the workflow: 28. And that's only meaningful in the context of the range (15-45). In other words, if every step received a 1, the best possible score would be 15 (there are 15 steps).

So, is 28 "good"? It's near the middle of the range, so it is "not terrible" but "not great". Any step rated a three is colored red and those are where the team should pay attention.

The overall score is colored red if ANY step in the entire workflow is red (after all, if there is one major usability problem, it jeopardizes the entire workflow).