Modelling the reliability of SPICE based assessments

Abstract

An area of major investigation in the SPICE trials is the reliability of assessments. In particular, their evaluation and improvement. Previous reliability studies in the trials have focused on evaluating reliability. In this paper we report on a study that focused on generating recommendations for improving the reliability of assessments through the construction of a model. The study attempted to identify factors that have an impact on reliability. Using data from three assessments, we constructed a model that explains some of the variation in the reliability of assessments. The two factors considered were the capability of processes and when ratings were made during an assessment (other factors such as assessor experience were held constant). Our model suggests that future assessment processes should have two contiguous phases in order to increase reliability. The first phase focuses only on data collection. During the second phase, ratings are made.