Commentary on the Article by Weissmuller and Damos

The article by Weissmuller and Damos provides an overview of how validation studies are normally conducted within the field of pilot selection, and it presents statistical approaches to predict pilot performance and a new way to develop selection system processes. A major focus of the article is to suggest new approaches when there is a shortage of pilot applicants. Validation studies are an important part of developing a selection system, and they also form the basis for evidencebased practice. A typical validation study is conducted on a military sample using pass–fail in basic training as a criterion. The validation studies are sometimes also based on a relatively small sample with no correction for statistical artifacts such as range restriction and criterion reliability (Martinussen, 1996). Weissmuller and Damos address several important aspects when conducting a validation study, including the problem with using multiple linear regression (MLR) when the statistical assumptions are violated, such as using a dichotomous criterion instead of a continuous one. This will undoubtedly lead to less precise estimated regression coefficients. We completely agree with Weissmuller and Damos that MLR is not appropriate when the dependent variable is dichotomous. In their review of statistical methods they state that “both MLR and LDA [linear discriminant analysis] assume multivariate normality for the independent variables.” This is a bit imprecise, as MLR assumes no particular distribution for the predictors. The normality assumption in MLR is that for a particular combination of values for the predictors, the dependent variable should follow a normal distribution. So the effect of this assumption is that residuals (or errors) should be normally distributed. The normal distribution assumption is automatically violated if MLR is used as a dichotomous dependent variable, as the errors for a given combination of predictor values only can take on one of two different values. Also, the homoscedasticity and linearity assumption is violated when using MLR on a dichotomous dependent variable. Therefore, violating the assumption of a continuous dependent variable in MLR gives, in reality, a violation of three of the basic MLR assumptions (see, e.g., Cortina, 2002). A critical point in any validation study is the choice of criterion. Ideally, it should be reliable, valid, and relevant to the organization. The use of pass–fail in training has been criticized for being a poor indicator of actual pilot performance, whether this is performance as a fighter pilot or as an airline captain. Sometimes, as indicated in the Weissmuller and Damos review, the pass–fail rate is very skewed, which leads to even bigger problems with documenting the predictive validity of the predictors. The pass–fail criterion could also be contaminated by irrelevant