A Modification on Intra Class Correlation Estimation for Ordinal Scale Variable Using Latent Variable Model

Abstract

Introduction: A common way for computing test-retest reliability is Intra Class Correlation which was developed for continuous variables. But it widely used to assess test-retest reliability in questionnaires with Likert scales. Most of the time consecutive numbers regarded as option labels of a question. If the probability of choosing options be the same, using this method is logic, otherwise it is not. Therefore, in this study a modified estimator of ICC is proposed to improve the estimation of ICC for ordinal scale by using latent variable model. Methods: In this method test-retest answers were considered as bivariate variables and cumulative Probit latent variable model was fitted. A simulation study with N=1500 replicates was conducted to compare the ICC estimations of Likert scale approach with a latent variable approach. Different sample sizes (n=20, 30) was generated with different correlation parameters. The simulations were repeated for questions with 3 and 5 options with different probability of selecting options of a question. After that the two approaches were run on Beck for suicidal ideation questionnaire. Results: In general the difference between Likert scale approach and latent variable approach were higher in 3 question options compared to 5 and also by increasing sample size and correlation between bivariate data, Root Mean Square Errors and bias were decreased. Assuming different probabilities for options, there was a considerably difference between Root Mean Square Errors, bias and standard deviation of estimation of ICC in two models. Using latent variable approach resulted less bias, SD and Root Mean Square Errors especially in lower sample sizes. Conclusion: Simulations showed when the probability of choosing options of a question are skewed, using this method reduced Root Mean Square Errors especially when the options are less. This method was affected more on standard deviation compare to bias of estimations.