Evaluating the Satisfaction of ABET Student Outcomes from Course Learning Outcomes through a Software Implementation

Abstract

Continuous improvement procedure to attain a certain level of program satisfaction is mainly based on the evaluation of ABET Student Outcomes (SOs) satisfaction in various courses. The satisfaction of a given SO is indicated by percentage of students obtaining a prescribed level of success in direct assessments. It is difficult for average instructors to design reliable assessments addressing the SOs because their focus in teaching is on Course Learning Outcomes (CLOs) that are related directly to the subject matter. To resolve this issue, a simple approach is presented to convert CLO-based assessment data to SO-based data through the CLO-SO map and a conversion formula. A software package “CLOSO” developed to implement this idea is described. The software automates the evaluation of CLO and SO satisfaction thereby enhancing the reliability of assessment data and saving instructor’s time significantly and generates summary reports for ABET course files. DOI: 10.4018/ijqaete.2012070102 22 International Journal of Quality Assurance in Engineering and Technology Education, 2(3), 21-33, July-September 2012 Copyright © 2012, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited. students demonstrate the attainment of eleven different SOs. These SOs are listed in Table 1. These outcomes may be modified or additional outcomes may be introduced to suit a particular program but most commonly the eleven SOs of ABET Criterion 3 are used without any modification. While SOs represent a set of general abilities to be attained by the students, the “Course Learning Outcomes” (CLOs) specified for all courses in a curriculum are specific to the content of the course and describe the course-related abilities students will acquire at the end of a course. It is essential that CLOs are distinct, non-overlapping and targeted to specific course-related skill levels (Felder & Brent, 2004). They also must be measurable through direct assessments. The questions asked in direct assessments like quizzes, homework and examinations, always target one or more of the CLOs. Table 2 shows an example of CLOs for a course. It must be noted that SOs are not targeted directly in any course. The courses target CLOs. The abilities represented by the SOs are attained by the students through the CLOs in various courses. To identify those SOs that are attained through the CLOs in a particular course, a CLO-SO map is required. For example, a CLO in the Computer Organization course is: “An ability to write programs in MIPS.” It is listed as CLO5 in Table 2. This CLO requires experiments to be conducted in the laboratory, therefore it maps to SO “b”: “An ability to design and conduct experiments, as well as to analyze and interpret data”. The same CLO also requires an ability to use software that facilitates programming in MIPS. Since this software is a modern engineering tool, CLO5 also maps to SO “k”: “An ability to use the techniques, skills, and modern engineering tools necessary for engineering practice.” The complete CLO-SO map for the course is shown in Table 3. CLOs depend upon the contents and the objectives of a course as outlined in the program syllabus therefore it is possible that the same course in two different institutions have different sets of Table 1. Student outcomes Engineering programs must demonstrate that their students attain the following outcomes: a. An ability to apply knowledge of mathematics, science, and engineering b. An ability to design and conduct experiments, as well as to analyze and interpret data c. An ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability d. An ability to function on multidisciplinary teams e. An ability to identify, formulate, and solve engineering problems f. An understanding of professional and ethical responsibility g. An ability to communicate effectively h. The broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context i. A recognition of the need for, and an ability to engage in life-long learning j. A knowledge of contemporary issues k. An ability to use the techniques, skills, and modern engineering tools necessary for engineering practice Table 2. Example of course learning outcomes A student who successfully fulfills the course requirements will have demonstrated: 1. An understanding of basic computer arithmetic algorithms 2. An ability to implement multi-cycle implementations of a computer instruction set 3. An ability to analyze a pipelined CPU 4. An ability to analyze and evaluate CPU and memory hierarchy performance 5. An ability to write programs in MIPS International Journal of Quality Assurance in Engineering and Technology Education, 2(3), 21-33, July-September 2012 23 Copyright © 2012, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited. CLOs and consequently the CLO-SO map may also differ. As described earlier, the basic components of an effective assessment plan include a program mission, a set of PEOs, a set of SOs and a curriculum with a set of prescribed CLOs with a CLO-SO map for each course. All of these components are to be assessed, evaluated and periodically modified to implement a sustainable continuous improvement process. An effective assessment plan is a combination of both indirect and direct assessments. Direct assessment is the most reliable and important method of assessment. It is done through assignments, projects, in-class tests, oral presentations etc. Indirect methods supplement the direct assessments. They include student and faculty surveys, exit interviews, employers’ surveys etc. (Bailie, Marion, & Whitfield, 2010). A step by step assessment procedure has been described by Blaha and Murphy (2001) and Browning and Sigman (2008) have presented a problem-based assessment approach that they applied at Drury University with encouraging results. In another publication (Wang, Schwartz, & Lingard, 2008), the authors have assessed two CLOs of a course in Software Engineering over a period of two years. Results of the assessment cycle have been analyzed. A learning assessment approach used in Computer and Information Systems Department at University of Houston has been presented in Yue (2007). The approach is vigorous in completing the entire assessment cycle and it enhances faculty participation. In an assessment plan, the data collected from direct assessments in various courses need to be analyzed and evaluated. This evaluation process is time consuming and burdensome. Several publications discuss this issue and attempt to present some solution. For example, Burge and Leach (2010) present a tool based on Excel macros to allow automatic determination of the degree to which individual students meet the learning objectives that indicate how well students meet the course objectives and program directives which is equivalent to evaluating the CLO and SO satisfaction. In Essa, Dittrich, Dascalu, and Harris (2010), a software tool – called ACAT has been presented which is a web-based tool to keep students’ records and generate various reports. Another web-based tool Web-CAT has been presented in Ringenbach (2010) that mainly manages students’ data providing a user-friendly interface with a number of options. However, it lacks the functionality of effectively assessing the CLOs and SOs required for ABET accreditation. In Haga, Morris, and Morrell (2011) and Morrell, Morris, and Haga (2009), a database management system has been presented for tracking course assessment data and reporting related outcomes for program assessment. A database management system has also been presented in Urban-Lurain et al. (2009) to store large assessment data of students. The system is useful in keeping track of historical data, for Table 3. Mapping of course learning outcomes (shown in Table 2) to student outcomes Course Learning Outcomes CE STUDENT OUTCOMES a b c d e f g h i j k