The Questionnaire for User Interaction Satisfaction(QUIS) is a usability testing tool designed to gauge computer user's subjective satisfaction with the computer interface. The QUIS contains a demographic questionnaire, an overall measure of satisfaction, and measures of user satisfaction in four specific interface aspects (screen factors, terminology and system feedback, learning factors, and system capabilities). The current study establishes some normal values for QUIS evaluations. Undergraduate students attending one of six classes in the AT&T Teaching Theater evaluated their experiences using a variety of software packages in the course of one semester. A new version of the on-line QUIS 5.5 ran in the Teaching Theater's Windowsª 3.1 environment and collected the students satisfaction data. The on-line version of the QUIS and the results it yielded are discussed.
The QUIS
The Questionnaire for User Interaction Satisfaction (QUIS) is a tool developed by a multi-disciplinary team of researchers in the Human / Computer Interaction Lab (HCIL) at UMCP (Chin, Diehl, & Norman, 1988). The QUIS was designed to assess user's subjective satisfaction with specific aspects of the human/computer interface. Previous efforts to build tools for evaluating the human /computer interface suffered from validation, reliability, and standardization problems (Ives, Olson, & Baroudi, 1983). The QUIS team successfully addressed these problems, creating a measure that is highly reliable across many types of interfaces.
The QUIS is currently licensed to 76 sites. Approximately 44% of these are commercial/industrial users, 32% are international education and research users, and 24% are domestic education and research users. Most sites use the QUIS in conjunction with a usability testing lab.
The QUIS contains a demographic questionnaire, a measure of overall system satisfaction along six scales, and hierarchically organized measures of four specific interface factors (screen factors, terminology and system feedback, learning factors, and system capabilities). Each area measures the overall satisfaction with that facet of the interface, as well as the factors that make up that facet, on a 9-point scale.
Even though the QUIS is a powerful tool for interface evaluation, the on-line version has been limited because of some interface issues. Previous versions of the QUIS have been laid out in a linear fashion, with one question per screen. This format fails to represent the hierarchical nature of the question sets, limiting continuity between questions. QUIS 5.5 presents related sets of questions on the same screen, lending continuity to the set, and reducing the amount of time subjects spend on navigation between questions. Users of the QUIS often avoid the on-line version because it didn't record specific comments about the system. These comments are often vital for usability testing. In response to this need QUIS 5.5 collects and stores comments on-line for each set of questions. Users were also frustrated with the format of the output of the QUIS, making it time consuming and confusing to analyze the QUIS data. QUIS 5.5 now stores data in a format that is easily imported into most popular spreadsheets and statistical programs. By far the most important change in QUIS 5.5 is it's new flexibility. Past versions required experimenters to use all of the questions in all of the areas. Most often, only a sub-set of the 80 questions are applicable to the interface being tested. QUIS 5.5 allows experimenters to select sub-sets of the QUIS that are of interest to them. This saves both subjects, and experimenters time and effort.
The AT&T Teaching Theater
The AT&T Teaching Theater provides an electronically supported learning environment for undergraduate education. Currently classes in a wide range of subjects (e.g., psychology, computer science, mathematics, business and management) are taking advantage of the networked computers and multi-media environment using both commercial and custom software.
Subjects
Students (55 males and 48 females with an average age of 26 years) completed the QUIS at the conclusion of a 12 week course in one of six courses offered in the teaching theater. Each course was taught by a different instructor on a different subject often using different software in the classroom.
Materials
Subjects completed the QUIS on personal workstations in the Teaching Theater. The short version of the QUIS 5.5 ran in Spinnaker Plusª environment operating in Microsoft Windowsª 3.1 operating system. Keyboard and mouse input devices were used to complete the questionnaire.
Procedure
Instructors of each of the six courses administered the QUIS in the last week of classes. Students were asked to evaluate the software they used in the Teaching Theater and were instructed in using the QUIS. Subjects completed the QUIS and their answers were written to a central file server.
Summary data for each question in the short version of QUIS 5.5 are presented in Table 1. Each of the six general satisfaction questions, and 22 specific questions are listed by question number.
Table 1.
User satisfaction ratings for the AT&T Teaching Theater. (Scores are on a 1 to 9 scale.)
The relationship between each questionnaire item is shown in Table 2. Questions within the general satisfaction measure are highly related, and specific questions also tend to be highly correlated with the general satisfaction questions. There are weaker correlations between specific questions from different interface areas.
Table 2.
The correlation matrix for all the measures in the QUIS 5.5 short version.
Mean scores for each class were also measured. Table 3 shows the overall score that each software package or group of programs received.
Table 3.
Scores for each class, listing the software used. Scores are on a 1 to 9 scale.
This study establishes a set of norms that can be used for the on-going evaluation of the AT&T Teaching Theater. As courseware is developed to take advantage of the unique strengths of the theater, improvement can be gauged in respect to past efforts. The QUIS is proving to be an excellent tool to guide the development of new software.
The hierarchical nature of the QUIS is also demonstrated here. The four separate interface factors measured in the QUIS are hypothesized to explain unique facets of total satisfaction. The six general satisfaction questions are intended to predict total satisfaction from a different approach. It is not surprising that the four specific interface factors each relate strongly to the general satisfaction questions because they both measure the same quantity, total satisfaction. Similarly, it is not surprising that questions from different interface factors are not highly related. Each factor measures a unique part of total satisfaction, and therefore questions from different factors measure different aspects of the system.
Despite the utility of these data to research in the AT&T Teaching Theater, the unique nature of the classroom must be taken into account before any generalizations can be made. First, the unique hardware found in the classroom may tend to facilitate, or deter the satisfaction found with the software. The large workstation screens used in the classroom may have tended to increase satisfaction in some instances. On the other hand, some applications, specifically Spinnaker Plusª, tended to run so slowly on the system that satisfaction measures may be overpowered by this single effect. Secondly, the fact that class instructors administered the QUIS may tend to bias the results to be more favorable than they should be.
Future research should focus on obtaining normative data for real world software packages and applications. Anchor points need to be established so that satisfaction can be compared in terms of contemporary competitors. And even though the QUIS 5.5 has been re-designed be itself more satisfying, the QUIS needs to stand up to it's own light and be subject to investigation. A meta QUIS study could improve the QUIS, as well as examine some issues of halo effect.
Partial support for this project was provided by a grant from AT&T Information Systems and the Computer Science Center at the University of Maryland. We wish to thank Walt Gilbert, Project Director of the AT&T Teaching Theater, and Ellen Yu, Project Manager, for their support. Finally, we wish to thank the instructors who allowed their students to participate in this study.