Tests, Grades, and New Criteria for Education
"I love every aspect of teaching except giving grades.." confessed Teacher A to Teacher B.
"But grading is a necessary part of education." Teacher B retorted. "The students need the feedback. What don't you like about it?"
Teacher A, who had a wonderful rapport with her students, responded, "Well, it takes time away from personal contact with the students who need help; it is tedious; and I hate the record keeping and averaging of grades. It puts me in the position of an adversary rather than a teacher."
Teacher C, who had been listening to the conversation in the teacher's lounge asked, "Well, what if technology could do it for you? Imagine a computer system that could do all the dirty work of testing and grading and leave the teaching to us. What would you have the computer do and what would you keep for yourself.?"
Teacher A quickly responded, "Everything."
"Everything? Even reading the papers and deciding how many "As" and "Bs" to give out?" asked Teacher C.
"Well, may be not everything. OK, I'd like to arbitrate grades and I'd like to add my professional evaluations of papers."
"OK, let's talk about it." said Teacher C, "Next week, there will be several focus group meetings of teachers to help in the design of an electronic system for learning evaluation. It turns out that the three of us are in the same group."
The focus group was facilitated by a graduate student from the state university working on a multi-million dollar grant to provide design specifications for a federally funded educational support system. The group included eight teachers, four students, and two administrators. The group was asked to generate as many requirements of a good electronic system for learning evaluation as they could think of and then prioritize them in terms of importance. The top ten requirements were as follows:
1. While objective tests can be automatically graded, essay questions are still very important and should not be eliminated for the sake of ease.
2. Grade sheets should be easy to use.
3. Students should have easy access to all of their grades and graded assignments.
4. Names on grade sheets should fill in automatically from class rosters; grades should fill in automatically from grading programs; and attendance should fill in automatically from the login system.
5. Grades should be accurate and reliable.
6. Students should be able to question grades and get an explanation from the teacher.
7. Grades should be based more on participation in activities, exercises in class, and assignments, and less on exams.
8. Scores should be easy to average; grade distributions should be easy to see; and grades should be easy to assign.
9. Tools should be available to aid in grading written assignments such as highlighting, cutting and pasting comments, and search for keywords.
10. A clear policy for assigning and averaging grades should be available to the students along with their current grades so that they know where they stand and what they need to do.
For many teachers the last thing that they want to do is assign grades. Nevertheless, grades are an essential part of the educational process. Tests and grades serve to motivate and evaluate. In the switched-on classroom, tests and grades serve the same functions. The difference is that in the switched on classroom, exams and grades should be more efficient, effective, reliable, and accurate. From the instructor's point of view they should be less time consuming, less of a burden, and more defensible than grades in traditional classrooms. From the student's point of view, they should be fairer, more accessible, and useful for redirecting learning activities.
As an integrated environment, the switched-on classroom should not only host on-line exams, quizzes, and other instruments of evaluation but it should also provide the means for recording, averaging, and disseminating the grades to the students and eventually to the school registrar. Evaluation and grading activities of the faculty should be facilitated by bringing to bear all of the tech-abilities listed in Chapter 3.
Over the past decade a number of programs have been developed for on-line test construction, test delivery, automatic scoring, and list and averaging of grades using spreadsheets. However, each program has generally been a piece in an unassembled jigsaw puzzle. Once again the objective of HyperCourseware is to integrate all of the parts from test construction to administration, from scoring to curving, and from record keeping to student feedback. Not only will integration make the job easier for the faculty, it will also improve the process for the student and even provide added features not possible in the hardcopy classroom.
In this chapter we will look at a number of grading and evaluation methods in the electronic environment and ways of making tests and grades meet the educational objectives in a more effective and efficient way. We will also look at some new techniques for evaluation and some of the implications of having grades on-line for the students.
10. 1 An Interactive Model of Evaluation Methods
Before considering specific methods of testing and evaluation in the switched-on classroom, it is important to reconsider once again the model of interaction presented in Chapter 5. In this model, interaction is between and among the instructor(s) and the student(s) within the course materials and course products. Testing and evaluation is in essence a meta-interaction since it involves an evaluative interaction about the learning interaction in the educational environment. As an evaluative interaction, the question is whether or not the learning interaction took place and whether it resulted in the desired course products. From a more traditional view, the question is whether the educational goals and objectives were met in course of instruction and the performance of the students. From the instructor's perspective, this question may be asked at the group level as to whether the necessary materials (e.g., topics and exercises) were covered adequately. Alternatively, it may be asked at the individual level as to whether each student covered the material, did the exercises, and demonstrated an adequate level of learning and comprehension.
Both questions involve tracking the paths from the objectives to the materials to the interactions to the products and back to the objectives as shown in Figure 10.1. When the instructors and/or the curriculum determine the educational objectives, we progress in a clockwise path through the schematic beginning and ending with the instructional group as shown in the top panel of Figure 10.2. This is an instructor/curriculum directed process. The student's choices are limited to the material and the interactions that serve to meet externally determined objectives. Such would be the case for instructional material for becoming a certified public accountant. The material is dictated by professional requirements, not by what the student thinks is important or interesting.
When the learner is in control and is self-directed, the student decides what he or she wants to learn from the material and in the end assesses the outcomes. In this case, it is a counterclockwise path beginning and ending with the student as shown in the bottom panel of Figure 10.2 This is a student/needs directed process. This would be the case in exploratory learning environments, upper-level and graduate seminars, and many adult education courses.
In either case, the electronic educational environment should be designed to track all aspects of what material should be covered, actually was covered, and what interactions took place both at the group and individual level. For example, the log of instructor activity in HyperCourseware shows what was covered during class sessions. The student logs show what they covered during study and what exercises they participated in. Either way, assessment of the learning activity and the outcome involves a tight coupling of the objectives, the materials, the interactions, and the products.
10.2 Evaluating the Process
Evaluating the educational process requires that the interactions of the instructor(s) and the student(s) with the materials and between each other be monitored, logged, and evaluated. What is of interest is not necessarily the outcome, but the steps taken in the course of producing the outcome. There are many things that can be monitored and logged in the electronic environment.
If one were interested in monitoring the studying process of the students, one could look at the following in order of increasing specificity: (a) the total number of logins to the materials and length of time logged in, (b) the total number of pages viewed and time per page, (c) the particular pages viewed and time per page, and (d) the completion of specific exercises, assignments, or answers on a particular page. While these measures might spot the student that is falling behind or help the student keep track of what he or she has covered, they are only a small part of the whole picture of the learning process. Figure 10.3, for example, shows that Lisa's profile for logins is very similar to the class average so at a superficial level she is keeping up with the rest.
At another level we could be interested in more complex paths through the material. Most courses are designed in a linear, sequential manner. The students are all lead through the same material in the same order. Hypermedia environments, however, allow for branching. Students my engage in local, in depth exploration; they may generate a new synthesis of divergent materials; or they may create their own unique, thematic tours through the media. In any case the path directly reflects the underlying process of learning that is not always captured in a static product such as a term paper. The software supporting the process may automatically capture the steps for review by the instructor. Alternatively, the student may selectively decide what part of process is stored for review by the instructor and what is not.
Furthermore, students may chronicle the process in a more traditional way by recording their observations of the process in an electronic journals. In some courses students are required to keep a week-by-week journal of their notes, activities, and reflections. Figure 10.4 shows one page of a student journal entry in HyperCourseware. Since the students know that the instructor also reads their journals, they are careful as to what personal information is recorded. They may also use the space for communicating with the instructor.
Records of group process are also useful for evaluation. Assigning a group project is useful when the task is beyond the capabilities of any one student as discussed in Chapter 8. However, the problem is that it is can be difficult to assign individual grades based on the group product without additional information on the process and individual contributions. If electronic records are kept of the contributions to a shared space, they can assist in providing an objective account useful for judging fairness of contributions. Journals of the group progress and individual assessments are also useful. Figure 10.5 shows the reflections of one student about a group project.
10.3 Evaluating the Products
In Chapter 6 we discussed the use of an assignments tool that not only lists the assignments but also provides a workspace in which the student can develop the material to be submitted. Once it is submitted, the completed assignment is copied to a directory that only the instructor can access. Grading written assignments follows more traditional approaches appropriate to the particular course being taught. What is unique about the electronic environment, however, is the ease of accessing submitted work, making annotations, recording a grade, and routing it back to the student.
In the paper based classroom assignments are passed in, handed in, or collected in some way. Instructors carry them off to their offices, to their homes, or somewhere else to grade them. Then they bring them back to class and hand them out or they are picked up by the students. All along the way there is a potential of loosing something. This is particularly true when there is any deviation from the routine and students hand in assignments early or late or at different locations and they don't find their way to the right pile or folder.
While it is not impossible to loose or misplace something electronically, a number of steps can be taken in the switched-on classroom to avoid the embarrassment of having lost a student's assignment.
Students retain a copy of all assignments in their personal workspace. When assignments are "handed in," the original copy stays in the student's personal workspace. Assignments may be written to the hand-in file space, but the original is retained in the student's personal workspace. The assignment is marked as completed and locked so that it cannot be altered by the student.
Collected assignments are stored in one common directory or folder. Separate copies of assignments and exams are stored in the instructor's workspace and may even be backed up on disk storage.
Copies are graded. Since grading involves changing the files that have been submitted, only copies of the files are graded, not the original. The instructor retains a copy of the graded assignments.
Returned assignments are stored under a slightly different name indicating that they have been graded. Graded assignments are copied to the student's workspace under a different name so that the students have both the original and the graded copy.
What is unique about the switched-on classroom is that multiple copies can be retained of the same documents. In the paper based classroom of the past, this was impossible. Even with the advent of the copy machine, it is impractical to copy and store all of the assignments. Electronic storage in education is a powerful and compelling force if managed properly.
Figure 10.6 shows a tool in HyperCourseware for accessing the completed assignments. The students' names or numbers are displayed in the column on the left. The particular assignment to be graded is selected on a pull down menu on the second column which then shows the points assigned. When one of the names is clicked, the field on the right shows the text that the student submitted.
From the instructor's perspective, grading is facilitated by the ability to easily retrieve assignments, the ability to annotate student work directly by typing or by cutting and pasting comments, and the ability to directly record grades on the electronic grade sheet. It is estimated that this can cut grading time by 25 to 50%. The implication is that if it is easier to grade assignments, instructors will have more time increase the quality of the interaction. More will said about the ease, accuracy, and quality of grading in a later section.
10.4 Types of Tests and Evaluation
Any type of test that can be given in the traditional classroom can be implemented electronically in the switched-on classroom. Simple "objective" tests such as true/false and multiple choice come to mind first because historically they have been implemented on computers for years. They are easy to grade by machine using a key. In the recent past such exams required special optical scanning sheets, #2 pencils, and care on the part of the students to blacken the correct spaces on the sheets. In the switched on classroom objective tests are even easier and more direct for the students. They can click on the correct alternative rather than circling it or transcribing it to another form. Exams can be automatically graded as soon as the student is finished. However, experience has shown that immediate feedback is not a good idea in a class setting since students may react in a positive or negative way to their grade and disrupt other students still taking the exam. In general, it is best to grade the exams after all of the students have completed it. If an item has been miss-keyed or a question was misleading and needs to be dropped, it can be done so before the grades have been disseminated. Furthermore, when all of the grades have been assigned, a distribution can be generated and disseminated to all of the students with their grades.
Exams with fill-in-the-blank items can also be graded automatically by using a key that lists acceptable terms along with a routine that matches strings independent of case and that allows minor discrepancies such as misspellings and word transitions.
In general objective tests are more difficult to construct. There is an art to writing good multiple choice and true/false questions. Since a significant effort is involved in writing objective tests, most instructors draw questions from published banks or re-use questions that they have written over the years. Either way there develops a database of questions that have a history of use. In the electronic educational environment it is possible to keep track not only of item use but also statistics on each question. Such a database can be used to ensure that items are not re-used too frequently and to eliminate items that are too easy, too difficult, or not diagnostic. This database can be limited to a particular instructor's use, a school's use, or use by the global community of instructors of a particular subject. As databases of questions grow, it will be necessary to code the items by a number of cues for retrieval. These might include: (a) author, (b) topic, and (c) a location reference to the textbook or course materials. Statistics collected on the questions may also be used for specifying the retrieval such as: (a) date last used, (b) an index of difficulty, and (c) an estimate of the amount of time required to answer the question.
As instructors select questions from the database they will want to screen the questions so as to cover the appropriate topics, to be at the appropriate level of difficulty, and to eliminate questions that overlap or interact in such as way to inadvertently reveal the correct alternative. The questions would then be imported into a module that presents the items and stores the student's responses.
In HyperCourseware the test module starts with a cover screen that automatically fills in the student's name but requires a student identification number for security purposes. The lower half of the cover screen is not seen by the student but has fields for the instructor to specify the starting and ending time of the exam and displays a log of all student access to the module. The log is used to verify that only the appropriate access was make to the test module. Figure 10.7 shows this screen.
The remainder of the module is composed of screens that display test items. For example, Figure 10.8 shows a multiple choice item. If the instructor is viewing the exam before the test, he or she can edit the items and click on an alternative to set the correct answer in the key and place a red circle around the box. When students are taking the exam, they click on the check box at the front of the correct alternative to place an X in the box and then go to the next item. Students are free to page through all of the items and to go back and change their answers. A bookmark can be placed on an item by clicking the "mark" check box. A list of marked and unanswered questions is displayed on the last screen. Students can return to any of these questions by clicking on the listed item. When either the instructor or the student views the module after the test, the answers are marked and locked. The check boxes of correct answers are circled in red if the student did not check it.
"Subjective" essay tests can also be given in the switched-on classroom. Subjective tests are easier to write but much more difficult to grade. They require either the teacher to grade them or require an intelligent system to scan for ideas and logical arguments and to assign a grade.
It is less likely to find databases of essay questions since they are relatively easy to write and less likely to be standardized. Nevertheless, it may be useful to store these questions in a database as well. This is particularly true if past student's answers to the questions are also stored. This allows the instructor to see the past success of each question and the range of answers from good to bad. It also provides the instructor with a prototype typical answer that can be used as a reference point for grading current answers. Furthermore, the database can provide a "best" answer that can be used to provide students feedback as to why points were deducted from their answer or can be used with cutting and pasting to annotate the student's answer.
Figure 10.9 shows a screen from an essay exam. The question is shown in the upper field and the student's answer in the lower field. When the instructor creates the exam, the question fields are unlocked. When the student takes the exam, the question field is locked and the lower field is unlocked for the student to enter his or her answer. When the instructor grades the exam the answer field is unlocked so that it can be annotated. Finally, when the exam in graded and open to the student all fields are locked.
Although a few students have worried about typing speed during exams, they have tended not suffer if adequate time is given. In general, however, most students do type and are able to produce longer and more coherent answers than if they were writing longhand. Furthermore, an increasing number of students have reported that they prefer to write exams on the keyboard rather than on paper because they are more comfortable with word processors than with pens or pencils.
Essay and short answer exams, of course, require the instructor to read and grade each one individually. Nevertheless, there are several ways in which the electronic medium can facilitate grading. One is the ability to reformat text and annotate text. While reading many of us will underline the key terms or serrate the key points. This can be done effectively using word processing commands. Annotations can be entered in a different color, style, or font. After grading several exams, one often finds that the same annotation is written over and over again. In the electronic medium, the instructor does not have to retype the same feedback each time, but merely re-copy it from a set of store comments. This procedure can reduce the time to grade a set of exams by as much as 30%. Moreover, it encourages more extensive and personalized feedback on each exam which in turn will encourage students to read and attend to the comments from the instructor.
10.5 Frequency of Evaluation
In education we can go from the extreme of giving a test after each lesson in a programmed learning system to giving only a midterm and a final in a college course. The frequency and type of evaluation depends on the material and the learning needs of the student. Not only should the switched on classroom support both extremes, but it should make it easier to bridge the gulf between the two. For example, in HyperCourseware, pop quizzes can be inserted into either the lecture material or at any point in the lesson plan for a day.
At the end of a lecture a short summary paper can be used to see what each student got out of the lecture. This can be used both for grading the student's ability to attend and extract the essential points of a lecture and for evaluating the effectiveness of the lecture in conveying the message.
Finally, major exams in HyperCourseware are handled through the exam scheduler. The exam scheduler helps the instructor to plan and implement exams and it helps the students to prepare themselves for upcoming exams. Figure 10.10 shows the listing of exams shown when the student clicks on the Exam Icon on the Home Screen.
The more assignments and exams in a course, the harder the students work, and the harder the instructor works grading and managing all of the grades. For this reason, instructors often fall short on the evaluation and feedback. It is too time consuming especially with large classes. How can the switched on classroom facilitate tests, assignments, and grading?
10.6 Grade Sheets
A typical grade sheet is constructed by listing the names of the students down the first column and the assignments and exams as labels of successive columns. Many instructors have transitioned from grade books to using computer spreadsheets for grade lists. However, they require entering the student names, entering the names of the assignments and exams, and entering all of the grades by hand.
Grade sheets in HyperCourseware are automatically generated from the list of students in the class roll and the lists of assignments in the Assignments Module and the Exam Scheduler. Grades can be automatically copied to the grade sheet when they are assigned during grading. A grade sheet in HyperCourseware is shown in Figure 10.11
The only thing that remains is how to average the grades to come up with a final grade. Some instructors assign points in such a way that they reflect the weight of each item. Others assign letter grades or percentages and then weight them when the grades are averaged. HyperCourseware provides either equal weighting or weights assigned by the instructor. Letter grades are transformed to percentages, averaged, and transformed back (e.g., A+ => 97.5%, A => 95%, A- => 92.5%, B+ => 87.5, etc.)
Grade distributions are also used in assigning letter grades. Consequently, distributions are generated for each grade in the grade sheet. They are also informative for the students to see where they fall within the distribution. Consequently, the distributions are disseminated to the students along with the grades in the Grades Module. Figure 10.12 shows a distribution generated for one exam. The screen also allows the instructor to add comments about the exam for the students to view.
10.7 Privacy , Efficiency, and Accuracy of Grades
In the paper based classroom graded papers and tests are often handed back folded or upside down in an attempt to keep the grades private. But invariably someone's prying eyes see a grade. Students with high marks are elated and may disrupt the class. Students with low marks try to hide their expression as they deal with disappointment and frustration in a group setting. Grades are sometimes posted outside the classroom by student id number depending on the school's policy. Again, it is fairly public and somewhat humiliating having your grade posed even by an id number.
In the switched-on classroom, grades are electronic, private, and ubiquitous. Once an exam is graded, the mark goes into the database and is disseminated to the students personal workspace. Figure 10.13 shows the grades listed in the Grades Module that can only be viewed by the student account holder. Grades can be as private as the student wishes. Grades are displayed whenever and wherever the student wishes to see them. One student wrote in an evaluation of HyperCourseware, "I like being able to see my grades with complete privacy."
Each time a grade is recorded and transferred from one record to another there is a chance for error. Typically, a grade is written on the exam or assignment. Then it is recorded on the grade sheet. Then the grades are tallied by entering them into a calculator. The total is recorded on the grade sheet. Other transformations finally result in a final grade which is then copied to a list to be posted and at many universities copied to a grade submission form using a number 2 pencil. Needless to say, errors abound.
In the switched-on classroom, electronic grades take advantage of the copiability and transformability of information. When a grade is assigned or automatically generated is it recorded on the exam, in the grade list, and disseminated to the student without having to re-enter it. One student wrote, "I trust the grades more because I know that I am looking at the same grade list that the instructor has."
From the instructor's perspective the efficiency and savings is immense. Take for example even a simple six question essay exam. Six marks, say on a 5-point scale, are written down for each exam. To write them down, total them on a calculator, and record the total on the exam and on the grade sheet requires 22 keystrokes or written digits. In HyperCourseware, it requires only 8 keystrokes, including two button clicks to total the marks and record the total on the grade list. It is not hard to see how copiability, transformability, and computability can realize a 60 to 70% savings in the amount of time recording and transcribing grades.
The main premise of this chapter was that testing and evaluation in the electronic classroom should be fundamentally different than in the traditional classroom. Evaluation of learning can focus on the process rather than the products. In electronic environments the system can unobtrusively monitor activities. Evaluation can be based on involvement in exploration, interaction with the materials, and interaction with others. Evaluation can also be based on products, such as assignments, reports, team projects, and exams. As it becomes easier to give assignments and exams, it is important for it also to be easier to grade materials and to manage the grades. News tools for grading and keeping track of grades are beginning to be developed. These tools should also provide students with better feedback and an accurate record of their grades and progress.
Exercises and Projects
1. Host a focus group of fellow student's and generate a list of good and bad aspects of current hard copy grading systems and personal experiences with those systems.
2. Host a focus group of fellow students and generate a list of fears and worst case scenarios of what could go wrong with grading and evaluation in a poorly designed switched-on classroom.
2. Although we may not like to talk about academic dishonesty, there are many new possibilities for cheating in the switched-on classroom. Divide the class into two groups. Have one group think of all the ways that students could cheat on exams in the electronic environment. Have the other group think of ways to stop cheating on exams. Compare the results and see if the second group was able to catch the first group.
[Table of Contents] [Chapter 9] [Chapter 11]