August 16, 1999

Navigational Issues in the Design of On-Line Self Administered Questionnaires

Kent L. Norman

Zachary Friedman

Kirk Norman

Rod Stevenson

Human/Computer Interaction Laboratory

University of Maryland

College Park, MD 20742

Introduction

Up until recent years surveys and questionnaires have been administered either by an interviewer or in paper-and-pencil form. The human interviewer method has the advantage in that a knowledgeable individual assists the interviewee with (a) the sequencing of items, (b) the interpretation of items, (c) the recording of the responses, and (d) the editing of errors. The disadvantage is that human interviewer (a) is expensive, (b) is not always conveniently available,(c) must be trained, (d) makes errors, and (c) can introduce bias into the interviewee's responses. Moreover, the interviewee is somewhat at the mercy of the interviewer who knows overall scope and content of the survey and reveals the items only one at a time.

The paper-and-pencil form has the advantage of being inexpensive. The disadvantages are that (a) items may be confusing without the aid of interpretation, (b) forms may be hard to follow from question to question and from page to page, (c) omissions and errors of various types can go without detection until it is too late to fix them, and (d) respondents may be less likely to start or once started to finish the questionnaire on paper than they would be under the personal persuasion of a human interviewer. However, paper-and-pencil forms do allow the respondent to browse the scope and content of the survey and the freedom to answer questions in any order.

Recently, computerized self-administered questionnaires (CSAQ) have become an attractive alternative to human interviewers and paper-and-pencil forms. This is particularly true as access to computer facilities increases and as distribution systems such as the World Wide Web (WWW) become more and more pervasive, accessible, and easy to use.

CSAQs have the potential of being even more inexpensive than paper-and-pencil forms and more interactive like human interviewers. They may be less expensive particularly on the WWW because one does not need to print or distribute a physical media. Furthermore, the responses are already in an electronic form and do not need to be physically sent back and coded. Rather, they can be electronically transmitted to a central server and automatically processed and analyzed. Moreover, CSAQs can be written in a number of interactive ways to assist in (a) sequencing of items and navigation of the forms, (b) giving definitions and clarification of terms, (c) recording responses, and (d) helping to avoid, detect, and correct errors at the point of entry or on completion of sets of items.

The goal of our research at the HCIL is to assist the U.S. Bureau of the Census in developing principles and guidelines for interactive CSAQs. This work will draw upon current theory and research in cognitive psychology, techniques of task analysis, and finally, empirical research conducted in our lab. The remainder of this report describes one such line of research. The study reported here focuses on the first fundamental problem/potential of interactive forms: the sequencing of items and the navigation of forms.

Before considering surveys and questionnaires on-line it is instructive to consider how people manage paper-and-pencil and interviewer assisted questionnaires.

First with paper and pencil surveys, the items are by necessity laid out in a fixed, sequential order on pages. The respondent can read through the items in the printed order and/or jump around to see the overall content of the survey. The respondent can easily get a idea of the length of the questionnaire and the direction in which the questions are going. Generally, the respondent will answer the questions in the order that they are printed, but in some cases skip sections and return to them later or jump ahead to questions that they want to complete right away.

In personal interviews, the interviewer controls the order of the items, presents only one question at a time, and waits for the response before going on to the next item. The interviewee can ask how long the survey will take and what it is all about; and generally the interviewer will answer these questions. Sometimes, the interviewee may want to skip questions and come back to them later. The interviewer might oblige but must take the responsibility of remembering to complete those items at a later point.

In either case, it must be remembered that as people are asked questions in a prescribed order or scan questions on a printed form, they must access the information from memory or from some external reference to answer the questions. Many factual questions may be answered from memory and, as such, will be subject to all of the problems associated with retrieval from human associative memory (ref). On the other hand, respondents need to consult external sources in order to answer factual questions. They may look at their driver’s license, birth certificate, medical records, checkbook ledger, company books, etc. Access to personal records, company records, and other sources will be subject to the physical accessibility and organization of these records as well as the respondent’s ability to search and retrieve information from these databases. Both internal and external sources are organized in ways that dictate or influence in some way the order of retrieval of information. When there is a match between the order of questions in a survey and the order of retrieval of information, processing is most efficient. When there is a mismatch, processing is less efficient and may result in errors. For example, if a person is asked to name the days of the week in temporal order, it is very easy. However, if they must list them in alphabetical order, it is not so easy (i.e., Friday, Monday, Saturday, Sunday, Thursday, Tuesday, Wednesday). Similarly, one is more likely recall the names of one’s children in chronological order or birth rather than alphabetically. However, records of employee files are more likely to be in alphabetical order rather than by date of hire.

Attitudinal and personal opinion questions can only be answered from one’s own personal experiential memory either directly or through a process of subjective assessment. This last type has been subject to much study. (Fischoff Liable Values..)

Attitudes are not likely to be stored or retrieved in fixed order although they may occur in clusters pertaining to subject (e.g., attitudes about foreigners) or linked by associations. Typically, items in questionnaires are grouped by topic. This may help to set the context for retrieval of attitudes and facilitate the process.

On-line surveys and questionnaires can be implemented in many different ways. In fact the design space of possibilities is enormous. Consequently, it is extremely important that interface development for computerized questionnaires and the software used to implement them be guided by reasoned principles and empirical research.

Just as surveys and questionnaires implemented in the print media or the personal interview must take into consideration the constraints and dynamics of their environments, the CSAQs must take onboard all of the issues and factors involved in the computer interface. The current design of computer interaction is focused on the graphical user interface(GUI). This interface is characterized by buttons, fields, windows, and mice. When surveys and questionnaires are implemented in GUIs, they be designed around the functions, facilities, and limitations of these tools. For example, windows that present surveys must be either scrolled or paged. The mouse can be used to select options to answer questions and buttons and scroll bars can be used to navigate through the questionnaire. Many other GUI tools and techniques are used to mediate input/output and control of the user interface (ref…)

Interfaces are modified to accommodate tasks and materials. In turn, materials are also modified to assimilate tasks and interfaces. For example, documents, whether manuscripts or surveys, are generally divided into smaller chunks for several reasons. They may be divided into meaningful sections to help define a mind set for thinking about a topic and answering questions of a particular vein. This will be referred to as "semantic partitioning." The smallest unit in a questionnaire or survey is the individual item, whether a question or rating scale. The respondent may be presented with items one at a time thereby focusing only on that item or presented with groups of items that provide a context or an allowance for comparison. All of these divisions are based on semantics of the questionnaire.

On the other hand, documents may be divided into pieces that fit within a physical space. Paging by length is a physical constraint imposed by paper size or by screen size. Page divisions interact with semantic divisions since they are usually out of synch. Page breaks may arbitrarily subdivide semantic partitions of a questionnaire. Worst of all is when page breaks fall within items of the questionnaire so that part of the question is on one page and the rest is on a subsequent page. If at all possible, this is to be avoided. Figure 1 shows a schematic of divisions of a survey.

Scrolling can be used to facilitate movement through sections that are longer than the screen length or window size. Scrolling avoids the problem of arbitrary page breaks in the middle of semantic sections or within the items themselves. The problem with scrolling is that it is generally hard to use as a navigational tool. It requires the user to mouse across the screen to a narrow hard to hit elevator with small buttons at the top and/or bottom. Research shows that scroll bars with additional paging functions are preferred (Norman, 1995; etc..). such as using the Page Up and Page Down keys.

When surveys are divided into sections, it can help with (a) semantic navigation, (b) thinking about a topic, and/or (c) accessing some type of information database. Semantic sections that fit within the screen size do not need to be further scrolled or paged; but those that don’t have the added problems of (a) requiring further navigation and (b) undermining the wholeness of a section by further subdividing it.

Navigation with pages and sections can be sequentially constrained by providing only forward and back buttons. Or it can be further enhanced with indexes and even search functions.

Consequently, the survey structure in conjunction with the interface presents a number of pros and cons associated with paging, sectioning, scrolling, and indexing. Much of this is mapped to performance and efficiency issues in terms of the time that it takes to complete questionnaires and the number of mouse clicks required to complete the questionnaire. Another portion must relate to accuracy and validity of the respondent information. Confusion in navigation may result in missing information from unanswered questions that were never seen or inaccurate information from misinterpreted questions due to faulty context.

Research and theory over the past 20 years in human/computer interaction has covered a number of issues in screen design and navigation of linear and sequential documents (ref…). Although much of this work relates directly to the design of CSAQs, there remains be a substantial portion of the interface dynamics of CSAQs that has yet to be developed.

The current study investigates different ways of partitioning surveys and the effects on navigation through the surveys. A long survey of 76 items was presented in four different ways: (a) one long, scrollable form, (b) divided into semantic partitions that require scrolling, (c) further sub-divided into screen sized pages, (d) presented item by item although several items were so long that they required scrolling.

In addition this study investigates the use of an index to sections and items for navigation. For each of the above four versions of the questionnaire, an index was added in a frame at the left hand side of the screen. For the long form, semantic partitioning, and screen partitioning versions, the index listed the semantic sections of the questionnaire. For the item version, the index listed Item 1 through Item 76.

The four versions and two index conditions (with or without) resulted in eight versions of the questionnaire. It should be noted that a number of other permutations could have been implemented. Semantic indexes could have also included the item numbers in those sections. This was not done in the present case so that respondents would have to find items by subject matter rather than by number. The item-based questionnaire could have listed semantic sections with the item numbers, but since the questionnaire was presented at the item level, semantic sections were not listed within the questionnaire itself.

In the first part of this study participants were asked to first fill out the survey on their own. It was expected that although there were some differences in terms of the number of mouse clicks required to navigate the questionnaire, there would no substantial differences in time to complete the questionnaire relative to the variability between individuals in their time to respond.

Moreover, it was expected that respondents would make little use of the indexes for navigation since they would be filling out the questionnaire in a more or less linear order. It was expected that participants would primarily use down scrolling within long sections and the "Next Section", "Next Page", and "Next Item" buttons on subdivided questionnaires.

In the second part of the study, respondents were asked to enter specific answers for two sets of eight items in the questionnaire. The first set of items were identified only by the text of time items (e.g., Enter $48 for the amount spend on trash collection in the past three months.). The second set of items were identified by item numbers along with the text (e.g., for Item 54 indicate that you would like to retire at age 65).

This last task is similar to the task of editing incorrect or missing information and the task of starting from an external database and entering information in a nonlinear order into the questionnaire. For the first set of items, it is expected that versions of the questionnaire that have semantic indexes will be faster and more efficient than versions that do not. For the second task, the version with the index of numbered items will be the fastest and most efficient.

Additional data on the navigational tools used and pages accessed should help to define strategies used by respondents and should provide information about how to design better questionnaire structure and navigational tools.

A generic questionnaire developed for research purposes by the Census was further customized for the present study. Items were grouped into 6??? Sections. Four versions of the questionnaire were generated. They were as follows: LF – one, long, scrolling from; SP – semantic partitioning into 6 sections; PP – page partitioning into 23 screens; SI – single items, 76 items. For the long form, respondents had to scroll through the form to see all of the items. In the partitioned forms, "Next" and "Previous" links were provided at the end of each partition to jump to the page section, page, or item depending on the particular version.

The forms were presented with or without a side index. For the long form (LF) and the semantic partitioned form (SP), the index listed the sections and subsections. The respondent could click on a topic in the index and the form would go to the top of that section. In the case of the single item version, the index listed the items from 1 to 76 without semantic labels. This resulted in 8 versions of the questionnaire listed in Table 1.

Two sets of eight comparable items were selected from representative areas throughout the questionnaire. When the respondent finished filling in the questionnaire on their own, they were asked to enter specific information for these items. For the first set of items only textual information was provided to help them find the item and enter the information (Set T). For the second set, numbers were provided that helped to locate the corresponding items in the questionnaire.

A questionnaire was also developed to assess the respondents subjective impressions of the interface. Ten items were rated in terms of agreement / disagreement on a 9-point rating scale.

All materials were generated in HTML for the World Wide Web. Pages were hosted on a Macintosh™ server running OS 8.5 and Webstar™ 4.0. Server logs recorded the pages accessed along with a time stamp. In addition, start and stop times were recorded in a separate file. Materials were presented to respondents using Netscape™ as the browser on a Macintosh computer with two monitors. The left monitor was a 15 inch display with 640 x 480 pixels. It was used for the presenting, the instructions, the start and stop buttons, and the information to enter into items in the questionnaire. The right monitor was a 17 inch display with 800 x 600 pixels. It was used for presenting the questionnaire. The system was configured such that the mouse moved between to monitors.

Respondents for this study were drawn from the university population and were paid $10 for their participation. Subjects ranged in age from __ to __ with a mean of __.__. Five males and 5 females participated each of the 8 conditions.

Participants were given instructions as to the nature of the task and filled out a consent form on the computer. Then they filled out a sort questionnaire on demographics asking their age, gender, and a self assessment of computer use and expertise. They were then randomly assigned to one of the versions of the questionnaire. They were asked to fill out the questionnaire on their own. They were asked to fill it out quickly but accurately and not to skip items unless specifically instructed to do so. When they finished, they clicked on a stop button to record the time. They were then shown information to enter for eight items and asked to go back into the questionnaire to change or enter that information. When finished they were again to click on a stop button. At that point the second set of eight items appeared and they were asked to go back and enter those answers and again click a stop button when finished. Finally, respondents were asked to complete the 10 item questionnaire on the interface for the main questionnaire. When done they were further informed as to the nature of the study, allowed to ask questions, and dismissed.

Figure 2 lists the time to completion for the 8 versions of the questionnaire. There were no significant differences in terms of time to completion. Number of pages accessed was of course a direct function of the version. Observations indicated that respondents filled out the questionnaires in a highly linear manner, scrolling down within the long form and scrolling down within the semantic partitions and advancing to next section, page or item. Respondents rarely used the index except for testing it out. No significant differences occurred between versions with or without the index.

Figure 3 lists the time to completion for the Text Set of items.

Figure 4 lists the time to completion for the Numbered Set of items.

Table 2 lists the mean ratings of agreements with the items on interface assessment for each of the eight versions.

See Table 2

Interface designs for questionnaires can vary in terms of division of forms into smaller and smaller parts. At one end are form-based questionnaires that try to present questionnaires as whole forms as they would appear on paper. At the other end are item-based questionnaires that present only one item at a time. The task nature of filling out a questionnaire may vary from a straight linear entry of answers to questions to a nonlinear jumping around. When entry is linear, the interface should support a smooth transition from one item to the next with minimal action on the part of the respondent. The eight versions did not differ substantially in this regard. Forward entry of items was supported either by scrolling down or clicking on the "Next" link.

When the task is nonlinear, the interface should support nonlinear access to items so that the respondent does not need to scroll or page through all of the items to get to the required item. In this regard the results indicated that without an index, scrolling was superior to many pages in the item-based version.

When semantic indexes were provided, they were used to access items when only semantic information was provided. Time to complete entry tasks was significantly shorter when indexes were provided than when they were not provided. Numeric indexes were of little use however, when only semantic information was given.

On the other hand when item numbers were given, the numeric indexes were highly efficient and task completion times were significantly shorter than all other versions.

Improvements….this suggests that best design will be one that accommodates the different questionnaire entry tasks from initial linear order to nonlinear access due to changes as function of edits, entry of missing information, or access to external files for information.

Semantic divisions help to set contexts during linear entry and aid in search for items on the basis on content.