{"title":"Search log analysis of user stereotypes, information seeking behavior, and contextual evaluation","authors":"Junte Zhang, J. Kamps","doi":"10.1145/1840784.1840820","DOIUrl":"https://doi.org/10.1145/1840784.1840820","url":null,"abstract":"Evaluation is needed in order to benchmark and improve systems. In information retrieval (IR), evaluation is centered around the test collection, i.e. the set of documents that systems should retrieve given the matching queries coming from users. Much of the evaluation is uniform, i.e. there is one test collection and every query is processed in the same way by a system. But does one size fit all? Queries are created by different users in different contexts. This paper presents a method to contextualize the IR evaluation using search logs. We study search log files in the archival domain, and the retrieval of archival finding aids in the popular standard Encoded Archival Description (EAD) in particular. We study various aspects of the searching behavior in the log, and use them to define particular searcher stereotypes. Focusing on two user stereotypes, namely novice and expert users, we can automatically derive queries and pseudo-relevance judgments from the interaction data in the log files. We investigate how this can be used for context-sensitive system evaluation tailored to these user stereotypes. Our findings are in line with and complement prior user studies of archival users. The results also show that satisfying the demand of expert users is harder compared to novices as experts have more challenging information seeking needs, but also that the choice of system does not influence the relative IR performance of a system between different user groups.","PeriodicalId":413481,"journal":{"name":"International Conference on Information Interaction in Context","volume":"49 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-08-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131295846","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Individual differences in gaze patterns for web search","authors":"S. Dumais, Georg Buscher, Edward Cutrell","doi":"10.1145/1840784.1840812","DOIUrl":"https://doi.org/10.1145/1840784.1840812","url":null,"abstract":"We investigate how people interact with Web search engine result pages using eye-tracking, to provide a detailed understanding of the patterns of user attention. Previous research has examined the visual attention devoted to the 10 organic search results, and we extend this by also examining how gaze is distributed across other components of contemporary search engines, such as ads and related searches. This provides insights about searcher's interactions with the \"whole page\", and not just individual components. In addition, we used clustering techniques to identify groups of individuals, with distinct gaze patterns. The groups varied in how exhaustively they examined the search results and in what regions of the search result page they paid most attention to (organic results vs. ads). These results further our understanding of how attention is distributed across increasingly complex search result pages, and how individuals exhibit distinct patterns of attention and interaction.","PeriodicalId":413481,"journal":{"name":"International Conference on Information Interaction in Context","volume":"46 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-08-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130553839","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
C. Lioma, Birger Larsen, Hinrich Schütze, P. Ingwersen
{"title":"A subjective logic formalisation of the principle of polyrepresentation for information needs","authors":"C. Lioma, Birger Larsen, Hinrich Schütze, P. Ingwersen","doi":"10.1145/1840784.1840804","DOIUrl":"https://doi.org/10.1145/1840784.1840804","url":null,"abstract":"Interactive Information Retrieval refers to the branch of Information Retrieval that considers the retrieval process with respect to a wide range of contexts, which may affect the user's information seeking experience. The identification and representation of such contexts has been the object of the principle of Polyrepresentation, a theoretical framework for reasoning about different representations arising from interactive information retrieval in a given context. Although the principle of Polyrepresentation has received attention from many researchers, not much empirical work has been done based on it. One reason may be that it has not yet been formalised mathematically.\u0000 In this paper we propose an up-to-date and flexible mathematical formalisation of the principle of Polyrepresentation for information needs. Specifically, we apply Subjective Logic to model different representations of information needs as beliefs marked by degrees of uncertainty. We combine such beliefs using different logical operators, and we discuss these combinations with respect to different retrieval scenarios and situations. A formal model is introduced and discussed, with illustrative applications to the modelling of information needs.","PeriodicalId":413481,"journal":{"name":"International Conference on Information Interaction in Context","volume":"174 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-08-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121268186","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Conceptualizing institutional repositories: using co-discovery to uncover mental models","authors":"Soo Young Rieh, J. Yang, E. Yakel, K. Markey","doi":"10.1145/1840784.1840809","DOIUrl":"https://doi.org/10.1145/1840784.1840809","url":null,"abstract":"This study investigates how people construct mental models of new information systems with which they have limited experience. Six different institutional repositories were used as the experimental systems for this lab-based co-discovery experimental study. Sixty subjects (30 pairs) were asked to complete search tasks based on a simulated work situations using an institutional repository. Subsequently, subjects were instructed to visually depict how they thought the institutional repository worked and then explain this to their partner. Our findings are based on these drawings, descriptors written on drawings, and audio-recordings of explanations and conversations. The results reveal that most of the subjects constructed mental models focusing on system operations and the design of the user interface. Few highlighted the interactivity between the system and the end user or presented a global-view of the system to show how it related to other search engines or databases. We found that the co-discovery method provides a viable research design to elicit people's mental model construction. The implications of the results for interactive information retrieval community and institutional repository community are discussed in terms of research design, search behavior, and user instruction.","PeriodicalId":413481,"journal":{"name":"International Conference on Information Interaction in Context","volume":"12 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-08-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126426671","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
P. Ingwersen, Marianne Lykke, Toine Bogers, Birger Larsen, Haakon Lund
{"title":"Assessors' search result satisfaction associated with relevance in a scientific domain","authors":"P. Ingwersen, Marianne Lykke, Toine Bogers, Birger Larsen, Haakon Lund","doi":"10.1145/1840784.1840826","DOIUrl":"https://doi.org/10.1145/1840784.1840826","url":null,"abstract":"In this poster we investigate the associations between perceived ease of assessment of situational relevance made by a four-point scale, perceived satisfaction with retrieval results and the actual relevance assessments and retrieval performance made by test collection assessors based on their own genuine information tasks. Ease of assessment and search satisfaction are cross tabulated with retrieval performance measured by Normalized Discounted Cumulated Gain. Results show that when assessors find small numbers of relevant documents they tend to regard the search results with dissatisfaction and, in addition, they obtain lower performance for all document types involved, except for monographic records.","PeriodicalId":413481,"journal":{"name":"International Conference on Information Interaction in Context","volume":"45 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-08-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127850492","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"On the potential search effectiveness of MeSH (medical subject headings) terms","authors":"Ying-Hsang Liu","doi":"10.1145/1840784.1840817","DOIUrl":"https://doi.org/10.1145/1840784.1840817","url":null,"abstract":"The creation of innovative techniques of document representation is critical to the development of effective information retrieval (IR) systems. In this paper, we report on the impact of state-of-the-art human indexing techniques, exemplified by MeSH (Medical Subject Headings) terms, in the document representation. We studied queries formulated by four different kinds of information seekers interactively using an experimental IR system: (1) search novices; (2) domain experts; (3) search experts and (4) medical librarians. The 3,442,321 documents came from the TREC 2004 Genomics Track document set. Effectiveness of retrieval was measured using the relevance judgments provided by TREC. Inclusion of MeSH terms in the document representation did not affect the effectiveness of queries with respect to precision and recall. Adding MeSH terms to the index did not have a positive impact on the effectiveness of queries formulated by different kinds of users. These findings contribute to our understanding of the associations between the users' cognitive space and the information objects in light of the principle of polyrepresentation.","PeriodicalId":413481,"journal":{"name":"International Conference on Information Interaction in Context","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-08-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131469734","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Performing document triage on small screen devices. part 1: structured documents","authors":"F. Loizides, G. Buchanan","doi":"10.1145/1840784.1840836","DOIUrl":"https://doi.org/10.1145/1840784.1840836","url":null,"abstract":"Document triage is defined as the rapid process by which information seekers make relevance decisions on a set of documents [1]. With the rising popularity of small screen readers such as Amazon's Kindle and the ubiquity of smartphones capable of displaying documents, we are faced with the challenge of facilitating information seekers with effective ways of searching for information using these small screen devices, while bypassing the space limitation affordance. In this paper we begin to explore how information seekers go about their document triage process on small screens beginning by looking at structured documents.","PeriodicalId":413481,"journal":{"name":"International Conference on Information Interaction in Context","volume":"19 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-08-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130313100","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Contextualizing user relevance criteria: a meta-ethnographic approach to user-centered relevance studies","authors":"Peiling Wang","doi":"10.1145/1840784.1840828","DOIUrl":"https://doi.org/10.1145/1840784.1840828","url":null,"abstract":"In this poster paper, we report a work-in-progress on contextualizing user relevance criteria using a meta-ethnographic approach. The purpose of the study is to derive higher-order interpretations of current studies to map user relevance criteria across contexts. The ultimate goal is to provide useful guidance to interactive information retrieval (IIR) design.","PeriodicalId":413481,"journal":{"name":"International Conference on Information Interaction in Context","volume":"257 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-08-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116480687","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Gérard Dupont, Aurélien Saint-Réquier, Sébastien Adam, Y. Lecourtier, B. Grilhères, S. Brunessaux
{"title":"A step toward an adaptive composition of query suggestion approaches","authors":"Gérard Dupont, Aurélien Saint-Réquier, Sébastien Adam, Y. Lecourtier, B. Grilhères, S. Brunessaux","doi":"10.1145/1840784.1840824","DOIUrl":"https://doi.org/10.1145/1840784.1840824","url":null,"abstract":"The problems of comparing search support tool in interactive information retrieval (IIR) and of selecting the right one have always been difficult due to the inherent dependency to users. Using an adapted evaluation protocol, we study in this paper different suggestion approaches. The results show that the performance are changing for different users and also during the search sessions. As a consequence, they also show that the selection of a particular support tool has to use new grounding. In this way, we propose a system that allows to combine independent suggestion mechanisms based on an analysis of user behavior and considering the search session time as a key factor instead of using only static rules.","PeriodicalId":413481,"journal":{"name":"International Conference on Information Interaction in Context","volume":"187 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-08-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122634123","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"What eyes can tell about the use of relevance criteria during predictive relevance judgment?","authors":"P. Balatsoukas, I. Ruthven","doi":"10.1145/1840784.1840844","DOIUrl":"https://doi.org/10.1145/1840784.1840844","url":null,"abstract":"This paper reports on the preliminary findings of a user study that explored how searchers fixate on information associated with different relevance criteria during the process of predictive relevance judgment. In order to address this objective a user study was conducted that involved the completion of questionnaires, use of eye tracking technology, talk aloud protocols and post-search interviews. As opposed to previous studies, the present research asked participants to search for real information needs that represented different search contexts (e.g. from searches about personal interest to academic related searches). This permitted the identification of several relevance criteria that naturally occur across different search contexts and the emergence of some fixation patterns, not observed before, associated to the use of these criteria. The paper concludes with a discussion of the impact and implication of this study in the wider context of relevance judgment and information seeking in context research.","PeriodicalId":413481,"journal":{"name":"International Conference on Information Interaction in Context","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-08-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114723570","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}