Comparison of type I error and statistical power between state trace analysis and analysis of variance

Abstract

State-Trace Analysis (STA) is a methodology for investigating the number of latent variables. Recently, a quantitative STA technique based on conjoint monotonic regression and double bootstrap method (STA-CMR) has been proposed. More discussion is needed on the type I error and the statistical power of this technique, as it adopts null hypothesis significance testing (NHST) to draw statistical inference. Because the results of STA are comparable with analysis of variance (ANOVA) in a three-factor experiment with linearity assumption, it is necessary to compare STA-CMR with ANOVA accordingly. This study investigated the type I error and the statistical power of STA-CMR and ANOVA in specific linear and nonlinear models using simulated data. Results demonstrated that both techniques were effective in the linear models, where ANOVA had a greater statistical power and STA-CMR had a more rigorous control of type I error. In the nonlinear models, although STA-CMR worked just as well as in the linear models, ANOVA completely lost its effectiveness. Besides, we found that the estimated type I error rate of STA-CMR was always smaller than the preset significance level in both linear and non-linear models. We suggest that the suppressed type I error rate may be caused by the bootstrap procedure, but the exact causes need further investigation. In conclusion, despite the suppressed type I error rate, STA-CMR can be a useful tool for determining the number of latent variables, particularly in non-linear models.