Shuqi Zhu, Xiaohui Xie, Ziyi Ye, Qingyao Ai, Yiqun Liu
{"title":"利用大脑信号比较点相关性判断和成对相关性判断","authors":"Shuqi Zhu, Xiaohui Xie, Ziyi Ye, Qingyao Ai, Yiqun Liu","doi":"10.1002/asi.24936","DOIUrl":null,"url":null,"abstract":"<p>How to collect relevance judgment has long been an important problem in Information Retrieval (IR). A popular method is to collect relevance judgment in a point-wise manner, in which assessors examine and give an absolute relevance score for each item independently of the others. As an alternative, pair-wise relevance judgment, also named preference judgment, allows an assessor to compare two items side-by-side and express their preference for one over the other. Previous work has explored the differences between these two paradigms of relevance judgments from many different aspects. Most of these works are conducted through explicit/implicit feedback. However, few works investigate the underlying neurological mechanisms of the two paradigms. In this paper, we conduct a lab study to investigate and compare point-wise and pair-wise relevance judgment in image search scenarios. We study the neurological mechanisms of the two paradigms through an event-related potential (ERP) analysis of the users' brain signals while viewing images during a search process. We have obtained several observations, such as search engine users tend to pay more attention to preferred items in the point-wise paradigm but unpreferred items in the pair-wise paradigm. Furthermore, we test the adoption of brain signals as implicit feedback for predicting pair-wise relevance judgment, highlighting the feasibility of leveraging brain signals to understand users' relevance judgments.</p>","PeriodicalId":48810,"journal":{"name":"Journal of the Association for Information Science and Technology","volume":"75 9","pages":"957-971"},"PeriodicalIF":2.8000,"publicationDate":"2024-06-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Comparing point-wise and pair-wise relevance judgment with brain signals\",\"authors\":\"Shuqi Zhu, Xiaohui Xie, Ziyi Ye, Qingyao Ai, Yiqun Liu\",\"doi\":\"10.1002/asi.24936\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p>How to collect relevance judgment has long been an important problem in Information Retrieval (IR). A popular method is to collect relevance judgment in a point-wise manner, in which assessors examine and give an absolute relevance score for each item independently of the others. As an alternative, pair-wise relevance judgment, also named preference judgment, allows an assessor to compare two items side-by-side and express their preference for one over the other. Previous work has explored the differences between these two paradigms of relevance judgments from many different aspects. Most of these works are conducted through explicit/implicit feedback. However, few works investigate the underlying neurological mechanisms of the two paradigms. In this paper, we conduct a lab study to investigate and compare point-wise and pair-wise relevance judgment in image search scenarios. We study the neurological mechanisms of the two paradigms through an event-related potential (ERP) analysis of the users' brain signals while viewing images during a search process. We have obtained several observations, such as search engine users tend to pay more attention to preferred items in the point-wise paradigm but unpreferred items in the pair-wise paradigm. Furthermore, we test the adoption of brain signals as implicit feedback for predicting pair-wise relevance judgment, highlighting the feasibility of leveraging brain signals to understand users' relevance judgments.</p>\",\"PeriodicalId\":48810,\"journal\":{\"name\":\"Journal of the Association for Information Science and Technology\",\"volume\":\"75 9\",\"pages\":\"957-971\"},\"PeriodicalIF\":2.8000,\"publicationDate\":\"2024-06-18\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of the Association for Information Science and Technology\",\"FirstCategoryId\":\"91\",\"ListUrlMain\":\"https://onlinelibrary.wiley.com/doi/10.1002/asi.24936\",\"RegionNum\":2,\"RegionCategory\":\"管理学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"COMPUTER SCIENCE, INFORMATION SYSTEMS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of the Association for Information Science and Technology","FirstCategoryId":"91","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1002/asi.24936","RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}
Comparing point-wise and pair-wise relevance judgment with brain signals
How to collect relevance judgment has long been an important problem in Information Retrieval (IR). A popular method is to collect relevance judgment in a point-wise manner, in which assessors examine and give an absolute relevance score for each item independently of the others. As an alternative, pair-wise relevance judgment, also named preference judgment, allows an assessor to compare two items side-by-side and express their preference for one over the other. Previous work has explored the differences between these two paradigms of relevance judgments from many different aspects. Most of these works are conducted through explicit/implicit feedback. However, few works investigate the underlying neurological mechanisms of the two paradigms. In this paper, we conduct a lab study to investigate and compare point-wise and pair-wise relevance judgment in image search scenarios. We study the neurological mechanisms of the two paradigms through an event-related potential (ERP) analysis of the users' brain signals while viewing images during a search process. We have obtained several observations, such as search engine users tend to pay more attention to preferred items in the point-wise paradigm but unpreferred items in the pair-wise paradigm. Furthermore, we test the adoption of brain signals as implicit feedback for predicting pair-wise relevance judgment, highlighting the feasibility of leveraging brain signals to understand users' relevance judgments.
期刊介绍:
The Journal of the Association for Information Science and Technology (JASIST) is a leading international forum for peer-reviewed research in information science. For more than half a century, JASIST has provided intellectual leadership by publishing original research that focuses on the production, discovery, recording, storage, representation, retrieval, presentation, manipulation, dissemination, use, and evaluation of information and on the tools and techniques associated with these processes.
The Journal welcomes rigorous work of an empirical, experimental, ethnographic, conceptual, historical, socio-technical, policy-analytic, or critical-theoretical nature. JASIST also commissions in-depth review articles (“Advances in Information Science”) and reviews of print and other media.