Measuring and understanding user reactions to computer software is important to many who are creating new services and programs, evaluating older versions, or making choices between similar products for certain applications. While the evaluation of a system's accuracy is fairly straightforward, the assessment of the user's satisfaction with the human- computer interface is a subjective and complex question. A multi- disciplinary team of researchers at UMCP has developed an instrument that evaluates user satisfaction with the human-computer interface aspect of other software packages and computer systems.
"Although a system may be evaluated favorably on every performance measure, the new system may not be used very much if the user is dissatisfied with the system and its interface," said Dr. Kent L. Norman, of the UMCP Department of Psychology and the Human-Computer Interaction Laboratory (HCIL west). "It's not just the utility of the program that matters, it's also the user's acceptance of the system -- subjective satisfaction -- that is a crucial measure of a program's success."
Norman and a team of researchers spent four years developing first a paper questionnaire and then a computerized questionnaire for assessing user's attitudes and subjective satisfaction with a system, especially the user's evaluation of the human computer interface. The instrument, the Questionnaire for User Interface Satisfaction (QUIS), is now available for non-exclusive licensing to companies that want to test user reactions to their software programs, and is in use world-wide.
"The QUIS does two things. It taps the overall subjective reaction of a user to an on-line computer system, and it is a diagnostic of the strengths and weaknesses of a system, " said team member Ben Shneiderman of the Computer Science Department and Head of the HCIL. "It assesses such things as satisfaction with the display of graphics, readability, reliability, understandability, and other features."
Many efforts have been made to create evaluation methods for computer systems, but most had problems with validation, reliability, sample size, or inability to account for subjective aspects of user reaction. The team that developed the QUIS instrument studied the past approaches to system evaluation and the chronic weaknesses that plagued early attempts. Their research, and the QUIS product, focuses on addressing those concerns with special attention given to both the general and specific subjective satisfaction with the system.
The QUIS has proven to have high reliability with low variability, which gives accurate information to companies seeking to evaluate user reaction to new products. Analysis of responses describes the assessment of overall reaction to the system, evaluation of the screen presentation, the terminology and system information provided, learning the system, and system capabilities. The QUIS is typically offered to users after they have completed a session of work with a particular system or program. The resulting statistical evaluation file gives feedback to system designers for product improvement.
For example, the QUIS has been in use for several years on the UMCP campus and hundreds of students have provided information on software available through campus systems. Of course, the students also served to help test the accuracy and reliability of the test itself. An important application of the QUIS at UMCP was part of a cooperative effort to help a Maryland small business through the Maryland Industrial Partnership program (MIPS) of the Engineering Research Center. A small company, Custom Command, which created a complex home automation system used the QUIS to test the human reaction to the computer interface with their program, which was driven by a touch screen. "The results of the QUIS evaluation gave a lot of useful information to the business that created the home automation system. The assessment led to the redesign of the screens and features, and greatly enhanced user satisfaction with the product that helps manage home lights, security, and environment," Norman said.
The QUIS is being offered through non-exclusive license agreements, and is already in use at a number of companies seeking to obtain overall satisfaction measures for their products, whether those be new products or new versions that the QUIS can evaluate as compared to older versions.