Do We Teach Useful Statistics for Performance Evaluation?

Abstract

Basic topics from probability and statistics -- such as probability distributions, parameter estimation, confidence intervals and statistical hypothesis testing -- are often included in computing curricula and used as tools for experimental performance evaluation. Unfortunately, data collected through experiments may not meet the requirements of many statistical analysis methods, such as independent sampling or normal distribution. As a result, the analysis methods may be more tricky to apply and the analysis results may be more tricky to interpret than one might expect. Here, we look at some of the issues on methods and experiments that would be considered basic in performance evaluation education.