Description, scenario, and profiles of the eight user-reviewers.

The tests took place in our usability lab in Waltham, between April 16th and April 24th. We allowed reviewers to look at the Good Documents site at the very beginning of a two-hour general review of the usability of Trellix 2.0.

Your questions were icebreakers within a larger review

The full review was elaborate, involving both actual software and prototypes (in paper and online). We start such reviews with an "icebreaker": a set of questions that let reviewer practice expressing strong opinions without putting pressure on them to "perform". This is often a Trellix Viewer reading task, comparing different types of documents to judge which are more effective. Your site offered plenty of things to comment on, and it also introduced the problem space that Trellix attempts to address, so its review was an excellent icebreaker. We would be happy to include this segment, or another with follow-on questions, in our upcoming reviews.

After the first reviewer, I revised the scenario

The first review (of eight) brought up problems with my review design which were addressed for the subsequent sessions. The questions listed in this (and all parallel) reports are the questions used for the group of seven. In the case of Good Documents, the modifications were minor: I changed the wording of question 3 so that I didn't ask explicitly for them to "find the tour."

Day-long gaps between reviews were less than optimal

Trellix build issues unrelated to your site caused one postponement over a week-end, and many of the reviewers could only visit our lab in the evenings. As a result, the review extended over nine days: I try to avoid this wherever possible.

In particular, because you were updating your site on a daily basis during this period, its design and contents changed subtly over the course of the test. However, even feedback on parts of the site that later changed can be of interest in general terms, so I'm simply reporting all the feedback I got.

The scenario was open-ended, but we used direct questions within it

For this task, I thought it would be useful to try to approximate the way the reviewers would actually work with the site. We tried to achieve this in two ways:

Each used his or her preferred Win32 browser. I encouraged them to stop wherever they would normally have stopped, if they'd encountered the site in their own browsing.

This latter technique has worked well with our Quick Tour: "If you would usually do the Quick Tours for a new products you were trying out, then let's do it. If not, let's skip it." I think it worked well here too (with the possible exception of the first reviewer, where you and I both sat in the lab during the review: I worry that this reviewer might have been a bit more polite than the others, due to concern about hurting your feelings).

Typically, the questions opened up discussion, and the reviewers then surfed through the site. As they clicked on each link - when I remembered to do so - I used a combination of UIE's technique of asking them why they were following each link, and (if that didn't yield enough data) InContext's technique of questioning with hypothesizing (e.g., "I'm guessing that you went back because that wasn't what you expected…?) and letting them correct my hypotheses.

As usual, detailed review of the transcripts reveals some areas in which my administrator-technique can continue to improve.

Questions

[Blue: actual questions cut in from the review booklet. Alignment and spacing are slightly different. All these questions were on one single page.]

Part I: Good Documents

First, we'd like your comments on Dan Bricklin's new web site, www.gooddocuments.com.

1. Open your preferred browser, either Internet Explorer or Netscape.

2. Access the site. What seems to be its purpose?

If this is not the type of site you'd normally be interested in, stop now. Otherwise, read any portions of the site that interest you.

Any general comments for Dan?

3. One part of "gooddocuments" explains how the site itself illustrates its own recommendations.

Is it clear how to find this part? Is it clear how to use it?

Do you have other comments?


Reviewer profiles

Four reviewers were information designers from the courseware department (that is, the department focused on content) of a web design studio serving corporate clients. One reviewer from this group was the department's manager. The studio does not have an "official" HTML editor: these reviewers used mostly Textpad (sp?) and HomeSite at work, but had used a number of other editors.

Two reviewers were experienced technical writer/editors, who had broad cross-platform experience with a variety of tools and information design problems (and their historical/canonical solutions).

Two reviewers were "engineers who do a lot of writing," who were primarily familiar with desktop applications (the Word and PowerPoint apps within MS-Office 97). One is currently serving as CTO for two projects being merged, and is writing planning documents related to this; the other consults on the design of real-time systems and often writes or revises manuals for the systems' operators.

The reviewers from the web studio worked in groups of two (co-discovery). I am comfortable that I got feedback from both reviewers, both times. The other reviewers worked alone. In addition to the observers behind the glass, there was an additional single observer in the room with us for one complete session and portions of two others.

This group of 8 reviewers was diverse in terms of sex and age. It was not ethnically diverse. Based on their questionnaires, jobs, and responses, I would call these reviewers "extremely computer savvy" relative to the general population, and "at least moderately sophisticated" relative to the average browser-of-the-Internet. We don't yet capture data that would let us be quantitative about this relative to other studies: these are gut feelings.

Basically, the reviewers who help Trellix at this point in our product's development fall into two groups: "experienced online writers" and "inexperienced online writers". Due to my needs for other parts of the review, this was an all-experienced group. I have no data (and no gut feeling either) as to whether this would make them more or less critical, but I just want to record it.