Evaluation of multivariate visualizations: a case study of refinements and user experience

Multivariate visualization (MVV) aims to provide insight into complex data sets with many variables. The analyst's goal may be to understand how one variable interacts with another, to identify potential correlations between variables, or to understand patterns of a variable's behavior over the domain. Summary statistics and spatially abstracted plots of statistical measures or analyses are unlikely to yield insights into spatial patterns. Thus we focus our efforts on MVVs, which we hope will express key properties of the data within the original data domain. Further narrowing the problem space, we consider how these techniques may be applied to continuous data variables. One difficulty of MVVs is that the number of perceptual channels may be exceeded. We embarked on a series of evaluations of MVVs in an effort to understand the limitations of attributes that are used in MVVs. In a follow-up study to previously published results, we attempted to use our past results to inform refinements to the design of the MVVs and the study itself. Some changes improved performance, whereas others degraded performance. We report results from the follow-up study and a comparison of data collected from subjects who participated in both studies. On the positive end, we saw improved performance with Attribute Blocks, a MVV newly introduced to our on-going evaluation, relative to Dimensional Stacking, a technique we were examining previously. On the other hand, our refinement to Data-driven Spots resulted in greater errors on the task. Users' previous exposure to the MVVs enabled them to complete the task significantly faster (but not more accurately). Previous exposure also yielded lower ratings of subjective workload. We discuss these intuitive and counter-intuitive results and the implications for MVV design.