The Role of Direct Manipulation of Visualizations in the Development and Use of Multi-level Knowledge Models

Abstract

The proliferation of touch sensitive display screens has created a new generation of human-computer interaction styles which are so natural and common that even the youngest of users now perceive ordinary static media like a glossy magazine as a broken iPad. The volume of users who expect to be able to pinch, grab, twist and manipulate images on screen is rapidly growing; they drive a renewed interest in developing, assessing, and delivering new direct manipulation systems. Our premise is that one can exploit new technologies to develop new repertoires of direct manipulation, but with increasing pressure to provide semantically-coupled direct manipulation methods to experiment with computational information models. We develop this premise by noting highlights in the evolution of direct manipulation interfaces, and suggest that their selection and deployment can be tailored as visual experiments to debug and extend more complex computational models of information systems and processes. These systems and processes include those of natural systems such as arise in systems biology (e.g., modelling multiple levels of protein structure), but also in "unnatural" systems such as in the identification of hubs and authorities in artificial systems like the World Wide Web (WWW). The immediate consequence of our premise suggests that the design of direct manipulation tools should proceed with the semantics of the modelled systems in mind, so that each users' manipulations provide a new perspective on the concept of "data mining" of large data sets. This will allow users to not just expose implicit relationships, but to incrementally combine explanatory and exploratory investigation by direct manipulation, to adjust and improve the computational knowledge models that emerge from the underlying data.