Stats & Maths & Unicorns

This chapter covers basic statistical concepts. Most statistics relate to hypothesis testing, and others to variable selection and model fitting. The name is because an exact match between a theoretical and empirical distribution is as rare as a unicorn. (1) Dispersion—measures of random variations—variance and its inflation factor, covariance and correlations {Pearson’s product-moment, Spearman’s rank order}, and the Mahalanobis distance. (2) Goodness-of-fit—do observations match expectations? This applies to both continuous dependent variables {R-squared and adjusted R2} and categorical {Pearson’s chi-square, Hosmer–Lemeshow statistic}. (3) Likelihood—assesses estimates’ goodness-of-fit to binary dependent variables {log-likelihood, deviance}, plus the Akaike and Bayesian information criteria used to penalize complexity. (4) The Holy Trinity of Statistics—i) Neyman–Pearson’s ‘likelihood ratio’—the basis for model comparisons; ii) Wald’s chi-square—for potential variable removal; iii) Rao’s score chi-square—for potential variable inclusion. These are all used in Logistic Regression.