Evaluating StackGenVis with a Comparative User Study

Stacked generalization (also called stacking) is an ensemble method in machine learning that deploys a metamodel to summarize the predictive results of heterogeneous base models organized into one or more layers. Despite being capable of producing high-performance results, building a stack of models can be a trial-and-error procedure. Thus, our previously developed visual analytics system, entitled StackGen Vis, was designed to monitor and control the entire stacking process visually. In this work, we present the results of a comparative user study we performed for evaluating the StackGen-Vis system. We divided the study participants into two groups to test the usability and effectiveness of StackGen Vis compared to Orange Visual Stacking (OVS) in an exploratory usage scenario using health-care data. The results indicate that StackGen Vis is significantly more powerful than OVS based on the qualitative feedback provided by the participants. However, the average completion time for all tasks was comparable between both tools.