Comparing Analytic and Mixed-Approach Rubrics for Academic Poster Quality

Abstract

Objective

Although there has been great interest in rubrics in recent decades, there are different types (with different advantages and disadvantages). Here, we examined and compared the use of analytic rubrics (AR) and mixed-approach rubric (MAR) types to assess the quality of research posters at an academic conference.

Methods

A previous systematic review identified 12 rubrics. We compared 2 notable ARs (AR1 and AR2) with a newer MAR. Sixty randomly selected research posters were downloaded from an academic conference poster repository. Two experienced academicians independently scored all posters using the AR1, AR2, and MAR. The time to score was also noted. For inter-rater reliability of scores from each rubric, traditional intraclass correlations and modern/advanced Rasch measurement were examined and compared between AR1, AR2, and MAR.

Results

The scores for poster quality varied using all rubrics. For traditional indexes of inter-rater reliability, all rubrics had equal or similar intraclass correlations using agreement, whereas AR1 and AR2 were slightly higher using consistency. The modern Rasch measurement showed that the single-item MAR reliably separated posters into 2 distinct groups (low quality vs high quality), which is the same as the 9-item AR2 but better than the 9-item AR1. Furthermore, the MAR’s single-item rating scale functioned well, whereas AR1 had 1 misfunctioning item rating scale and AR2 had 4 misfunctioning item rating scales. Notably, the MAR was quicker to score than the AR1 or AR2.

Conclusion

This MAR measured similar or better than 2 ARs and was quicker to score. This investigation illuminated common misconceptions that ARs are more accurate and a best use of time for effective measurement.