On Optimal Correlation-Based Prediction

Abstract This note examines, at the population-level, the approach of obtaining predictors of a random variable Y, given the joint distribution of , by maximizing the mapping for a given correlation function . Commencing with Pearson’s correlation function, the class of such predictors is uncountably infinite. The least-squares predictor is an element of this class obtained by equating the expectations of Y and to be equal and the variances of and to be also equal. On the other hand, replacing the second condition by the equality of the variances of Y and , a natural requirement for some calibration problems, the unique predictor that is obtained has the maximum value of Lin’s (1989) concordance correlation coefficient (CCC) with Y among all predictors. Since the CCC measures the degree of agreement, the new predictor is called the maximal agreement predictor. These predictors are illustrated for three special distributions: the multivariate normal distribution; the exponential distribution, conditional on covariates; and the Dirichlet distribution. The exponential distribution is relevant in survival analysis or in reliability settings, while the Dirichlet distribution is relevant for compositional data.