Nontraditional approaches to analyzing design of computer support and user evaluation

Abstract

While the potential benefit of additional measurement tools and techniques has been recognized in numerous domains, many questions remain about what to measure and how to get meaningful measurements. Although the majority of researchers seem to agree that user interaction metrics are helpful, the forms of these metrics causes considerable debate. The relationship between humans and computers (HCI) provides a rich area of study from both a theoretical and a practical perspective. Results from HCI research can contribute to theories of human behavior during complex tasks and often the results can be applied to the design of computer systems. An objective in coordinating this mini-track is to encourage a variety of papers that address the need for enhanced, nontraditional measures of performance to evaluate how people use interactive computer systems. Contributions to this mini-track include a wide range of topics as well as evaluation metrics. The first set of papers concerns the evaluation of student learning. Tsaganou, Cavoura and Grigoriadou present a methodology for the experimental construction of a student model of historical text comprehension that was based on a model of comprehension of historical narration (MOCOHN). They conduct an experimental study of students' text comprehension that demonstrated the effectiveness of their proposed student model. Chou employed content analysis to investigate the patterns of student to student interactions in distance learning environments. Vick, using experience gained in face to face classes, judges collaborative knowledge construction using goal-based scenarios in a synchronous computer-mediated communication situations. The second set of papers relies on physiological measurements to assess either skills or mental processes. Serra and Muzio explore the challenges of designing software to assist in the rehabilitation of people who have suffered traumatic brain injuries. They discuss suggestions for ways in which software can minimize the detrimental effects of impaired cognitive functions. Because of the unique characteristics of their patient population, the researchers employ an informal evaluation method and adapt their software to the patients needs. Ark, uses functional Magnetic Resonance Imaging (fMRI) as a tool to understand the cognitive ability of mental rotation. She summarizes findings from recent mental rotation studies and discusses the possible confounds between the reported results. King explores consistencies of eye-movement patterns by building a neural network and using statistics to predict eye movement behavior in different searching tasks. The final two sets of papers involve guidelines for web based materials and the software architecture needed to support web based data collection. Sutcliffe proposes a three-phase model for web site evaluation. He defines usability as a trade-off between increasing the user's motivation and the costs of usability errors. His proposed heuristics for assessing the attractiveness of web user interfaces contrast the web sites of three airlines. The Nordbottens analyze the use of a national statistical web-database. They have collected one years worth of data to show how people use web data bases. Yatim studies the disorientation problems people have using WWW environments. She uses a measurement based on a psychological model of the disorientation problem and finds a task dependent relationship between the time users take and the disorientation they experience. Stelovsky and Aschwanden discuss how