Evaluating Designer Learning and Performance in Interactive Deep Generative Design

Abstract

Deep generative models have shown significant promise to improve performance in design space exploration (DSE), but they lack interpretability. A component of interpretability in DSE is helping designers learn how input design decisions influence multi-objective performance. This experimental study explores how human-machine collaboration influences both designer learning and design performance in deep learning-based DSE. A within-subject experiment is implemented with 42 subjects involving mechanical metamaterial design using a conditional variational auto-encoder. The independent variables in the experiment are two interactivity factors: (i) simulatability, e.g., manual design generation (high simulatability), manual feature-based design synthesis, and semi-automated feature-based synthesis (low simulatibility); and (ii) semanticity of features, e.g., meaningful versus abstract latent features. We perform assessment of designer learning using item response theory and design performance using metrics such as distance to utopia point and hypervolume improvement. The findings highlights a highly intertwined relationship between designer learning and design performance. Compared to manual design generation, the semi-automated synthesis generates designs closer to the utopia point. Still, it does not result in greater hyper-volume improvement. Further, the subjects learn the effects of semantic features better than abstract features, but only when the design performance is sensitive to those semantic features. Potential cognitive constructs, such as cognitive load and recognition heuristic, that may influence the interpretability of deep generative models are discussed.