Using Generalizability Theory to Assess the Reliability of Student Ratings of Academic Advising

Abstract

The coefficient alpha procedure has been used frequently to assess the reliability of student ratings of academic advising, despite the fact that in many cases, coefficient alpha is not an appropriate procedure for providing reliability information on this type of measurement data. Data were collected from 15 postsec ondary institutions' use of the Survey of Academic Advising, and a more advanced framework, that is, the generalizability theory, was used to analyze the reliability of the data. The results of the study demonstrate that the reliability of student ratings of advising can be estimated more appropriately and accurately by using the proce dures of generalizability theory. Advantages and disadvantages of the generalizabil ity theory approach in assessing the reliability of student ratings of advising are dis cussed in comparison with those of the coefficient alpha procedure. USING STUDENT RATINGS as an outcome assessment to evaluate the qual ity of advising programs is a common practice in many postsecondary educa tional institutions. The results of such assessment often play a critical role in decisions about the future of the individual advisor?decisions about promotions and tenure and about the advising program, such as whether or not to continue a specific program or service (Severy, Lee, Carodine, Powers, & Mason, 1994). To justify the credibility of a decision, the decisionmaker needs some assurance from the data that the same or similar results could be obtained if the same advi sors and advising programs were rated again under similar circumstances. In measurement theory terms, these ratings should have a desirable level of relia bility.