The Tasks

Each of the eight participants is, in turn, asked to accomplish the same sequence of tasks, and since most users begin to tire after an hour, the sequence is aimed for a one hour duration. Clearly, on a big site with thousands of pages, one hour doesn't get you very far, so it's crucial to choose the right tasks. A big transaction including lots of online forms could take up half an hour on its own.

Evaluations tend to divide into two types - broadbrush tests looking at multiple aspects of a site, but not in great detail, and more focussed evaluations covering just one element of the site but with great thoroughness

Choosing the right tasks is part of the skilled work of the testing company. The Usability Company often begins question formulation with online questionnaires added at an early stage to the live site. These help to identify areas where visitors may be experiencing problems, and are likely to be combined with path data from analysed Web logs.

Typical tasks include registration, finding specific information, entering a competition or finding and buying a specific product. The number of tasks is usually low - a dozen at most and sometimes just two or three. Practical examples of tasks for a mortgage site under test included - You currently have the following mortgage (details given), see how an online mortgage from this site would compare. And - Use this site to apply online for the mortgage described.

The exact phrasing of the tasks is also crucial, and it can take time to explain to clients why one particular phrase has the correct psychological implications while another suggested phrase would only produce low quality data. It's important that the phrasing doesn't give specific instructions on *how* to accomplish the tasks, only what end result is required.

Pre-Test Questionnaire

Before testing begins, users are given a pre-test questionnaire. This usually requests demographic information such as age, gender, and level of Internet experience, to confirm the user comes from the right demographic.

Test Laboratory

The user sits at a regular computer screen, with a video camera to pick up their facial expressions and a microphone to record speech. Their screen is captured by a separate, synchronised online video, including cursor movements. In practice most users ignore the camera and microphone after just a few minutes. It is possible to track eye movements too, but rarely necessary.

A dedicated laboratory is useful because of the amount of recording involved. It also helps to cut down on distractions. Apart from the user, the only other person in the room should be the usability specialist, who sits alongside and watches. If clients are present, they sit behind a one-way mirror in an adjoining soundproof room. Clients are encouraged to send staff along to watch the tests, because it helps them get more out of the results. They also get the opportunity to ask the specialist questions between tests.

The Specialist

The specialist/analyst who sits in on the test will almost certainly be a behavioural psychologist, with cognitive psychology skills (the process of learning and understanding) and knowledge of HCI (Human Computer Interaction). They will also be a usability expert, but it's likely that their background will be in psychology rather than site design. When testing, psychology is far more important than the rational mechanics of good information architecture, though it's clearly desirable to understand both.

This same specialist helps to formulate the tasks, runs the tests, analyses the video, writes the test report and makes the final presentation to the client.

The specialist generally watches and listens to the test user, but may decide to interfere if they feel that time is being wasted, even to the extent of suggesting how to complete a task. This is one of the skills of the specialist - judging when interference will increase the amount or quality of data, rather than reduce it.

Users are encouraged to think aloud, to articulate thoughts about the site. They may also be prompted for thoughts by the specialist - for example if they get stuck at a particular stage.

Allowing for questionnaires and breaks, a test schedule usually fits in four tests on one day, followed by four the next to complete the set of eight. The site under test shouldn't be changed over the two days, though there are amusing tales of clients behind the glass on their mobile phones getting live changes made to the site while a user is still part way through a task, or leaving the laboratory to go back to the office and completely redesign the live site between one test day and the next.

Post Test

The final element of each test session is for the user to fill in a post-test questionnaire. These are short and take less than 15 minutes to fill out. They include questions like - How easy was the site to use, would you be likely to use it in the future, which areas did you like, and so on. Many of the questions don't relate to the specific tasks that were performed, because that information should already have been picked up by monitoring the user at the screen.

Results

The client company gets the results as a video, a report, and a presentation. They don't generally want to watch the entire eight hours of testing, but will probably be interested in seeing repeated reactions to problems areas, which can be sewn together as a clip with a split screen showing the computer screen, including cursor, and the participant's face, along with their comments on the soundtrack.

The report will offer recommendations for actions in each problem area, so the client can take immediate action. It's surprising how much information can be generated by a handful of tasks taking one hour in total. A sample report had over 70 recommendations, and the mortgage application mentioned earlier generated even more.

Usability Testing in Practice
Usability Testing in Practice
ROI - Page 3