Dynamic decision-making paradigm for multi-modal information in a human–computer interaction perspective: Fusing composite rough set and incremental learning

Abstract

The knowledge contained in different modal information and their perspectives on the description of specific objects are different. With the development of data science and computer technology, uncertain decision-making with multi-modal, nonlinear, unbalanced, and incomplete characteristics has become a trend, posing a great challenge to traditional uncertain decision-making methods. In view of this, this paper proposes a dynamic decision-making paradigm for multi-modal information by fusing composite rough set and machine learning under the perspective of human–computer interaction. First, a multi-modal hybrid attribute information system (MHAIS) is constructed for incomplete multi-modal hybrid attribute information in numerical, textual and image modals, and several degradation scenarios of MHAIS are discussed. Second, to achieve multi-modal information fusion and reduce redundant attributes, multi-modal hybrid binary relationships are constructed for numerical, textual and image modals in MHAIS, and multi-modal composite rough set and its attribute reduction method is given. Then, uncertainty decision-making models conforming to different realistic decision-making situations are established from the perspectives of three multi-modal information fusion strategies, namely feature-level fusion, model-level fusion and decision-level fusion, respectively. Finally, the above process is extended to the dynamic decision-making process, and the corresponding incremental learning paradigm is given. An example study using real datasets of rheumatoid arthritis patients from Guangdong Provincial Hospital of Traditional Chinese Medicine and 10 public datasets is carried out in this paper to verify the scientific and superiority of the proposed method. On the one hand, this paper introduces the idea of human–computer interaction and different strategies of multi-modal information fusion into realistic uncertain decision-making problems, and on the other hand, it makes a new contribution of the integration of rough set theory and machine learning in the development of management science, decision science and computer science.