Pavel Braslavski, Vladislav Blinov, Valeriia Bolotova-Baranova, Katya Pertsova
{"title":"How to Evaluate Humorous Response Generation, Seriously?","authors":"Pavel Braslavski, Vladislav Blinov, Valeriia Bolotova-Baranova, Katya Pertsova","doi":"10.1145/3176349.3176879","DOIUrl":"https://doi.org/10.1145/3176349.3176879","url":null,"abstract":"Nowadays natural language user interfaces, such as chatbots and conversational agents, are very common. A desirable trait of such applications is a sense of humor. It is, therefore, important to be able to measure quality of humorous responses. However, humor evaluation is hard since humor is highly subjective. To address this problem, we conducted an online evaluation of 30 dialog jokes from different sources by almost 300 participants -- volunteers and Mechanical Turk workers. We collected joke ratings along with participants» age, gender, and language proficiency. Results show that demographics and joke topics can partly explain variation in humor judgments. We expect that these insights will aid humor evaluation and interpretation. The findings can also be of interest for humor generation methods in conversational systems.","PeriodicalId":198379,"journal":{"name":"Proceedings of the 2018 Conference on Human Information Interaction & Retrieval","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124786600","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Kevin Ong, K. Järvelin, M. Sanderson, Falk Scholer
{"title":"QWERTY: The Effects of Typing on Web Search Behavior","authors":"Kevin Ong, K. Järvelin, M. Sanderson, Falk Scholer","doi":"10.1145/3176349.3176872","DOIUrl":"https://doi.org/10.1145/3176349.3176872","url":null,"abstract":"Typing is a common form of query input for search engines and other information retrieval systems; we therefore investigate the relationship between typing behavior and search interactions. The search process is interactive and typically requires entering one or more queries, and assessing both summaries from Search Engine Result Pages and the underlying documents, to ultimately satisfy some information need. Under the Search Economic Theory model of interactive information retrieval, differences in query costs will result in search behavior changes. We investigate how differences in query inputs themselves may relate to Search Economic Theory by conducting a lab-based experiment to observe how text entries influence subsequent search interactions. Our results indicate that for faster typing speeds, more queries are entered in a session, while both query lengths and assessment times are lower.","PeriodicalId":198379,"journal":{"name":"Proceedings of the 2018 Conference on Human Information Interaction & Retrieval","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126272068","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Sandeep Avula, Gordon Chadwick, Jaime Arguello, Robert G. Capra
{"title":"SearchBots: User Engagement with ChatBots during Collaborative Search","authors":"Sandeep Avula, Gordon Chadwick, Jaime Arguello, Robert G. Capra","doi":"10.1145/3176349.3176380","DOIUrl":"https://doi.org/10.1145/3176349.3176380","url":null,"abstract":"Popular messaging platforms such as Slack have given rise to hundreds of chatbots that users can engage with individually or as a group. We present a Wizard of Oz study on the use of searchbots (i.e., chatbots that perform specific types of searches) during collaborative information-seeking tasks. Specifically, we study searchbots that intervene dynamically and compare between two intervention types: (1) the searchbot presents questions to users to gather the information it needs to produce results, and (2) the searchbot monitors the conversation among the collaborators, infers the necessary information, and then displays search results with no additional input from the users. We investigate three research questions: (RQ1) What is the effect of a searchbot (and its intervention type) on participants» collaborative experience' (RQ2) What is the effect of a searchbot»s intervention type on participants» perceptions about the searchbot and level of engagement with the searchbot' and (RQ3) What are participants» impressions of a dynamic searchbot? Our results suggest that dynamic searchbots can enhance users» collaborative experience and that the intervention type does not greatly affect users» perceptions and level of engagement. Participants» impressions of the searchbot suggest unique opportunities and challenges for future work.","PeriodicalId":198379,"journal":{"name":"Proceedings of the 2018 Conference on Human Information Interaction & Retrieval","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131589930","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Better Together: An Interdisciplinary Perspective on Information Retrieval","authors":"S. Dumais","doi":"10.1145/3176349.3176571","DOIUrl":"https://doi.org/10.1145/3176349.3176571","url":null,"abstract":"The success of information retrieval systems depends critically on both the ability of systems to efficiently and effectively retrieve information, and to support people in articulating their information needs and making sense of the results. This interdisciplinary, user-centered perspective on information systems motivated my early work on Latent Semantic Indexing (LSI), which sought to mitigate the disagreement between the vocabulary that authors use in writing and searchers use to express their information needs, and continues to shape my research today. Over the last two decades, search has become a core fabric of people's everyday lives, driven by advances in understanding context, natural language, and speech. I will illustrate how new capabilities in email and virtual assistants are driven by advances in both algorithms and user modeling. As we look forward to new types of information systems that anticipate information needs, interact via richer dialogs, and integrate physical and digital information, it is more important than ever to understand and support information seekers using interdisciplinary methods and perspectives.","PeriodicalId":198379,"journal":{"name":"Proceedings of the 2018 Conference on Human Information Interaction & Retrieval","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115731798","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
J. Dinneen, Banafsheh Asadi, I. Frissen, Fei Shu, C. Julien
{"title":"Improving Exploration of Topic Hierarchies: Comparative Testing of Simplified Library of Congress Subject Heading Structures","authors":"J. Dinneen, Banafsheh Asadi, I. Frissen, Fei Shu, C. Julien","doi":"10.1145/3176349.3176385","DOIUrl":"https://doi.org/10.1145/3176349.3176385","url":null,"abstract":"Many large digital collections are organized by sorting their items into topics and arranging these topics hierarchically, such as those displayed in a tree view. The resulting information organization structures mitigate some of the challenges of searching digital information realms; however, the topic hierarchies are often large and complex, and thus difficult to navigate. Automated techniques have been shown to produce significantly smaller, simplified versions of existing topic hierarchies while preserving access to the majority of the collection, but these simplified topic hierarchies have never been tested with human participants, and so it is not clear what effect simplification would have on the exploration and use of such structures for browsing and retrieval. This study partly addresses this gap by performing a comparative test with three groups of university students (N=62) performing ten topic hierarchy exploration tasks using one of three versions of the Library of Congress Subject Headings (LCSH) hierarchy: 1) the original LCSH hierarchy, acting as a baseline, 2) a shallower version of 1), and 3) a narrower version of 2). A quantitative analysis of measures of accuracy, time, and browsing shows that participants using the simplified trees were significantly more accurate and faster than those using the unmodified tree, and the narrower, balanced tree was also faster than the shallower tree. These results show that automated topic hierarchy simplification can facilitate the use of such hierarchies, which has implications for the development of information organization theory and human-information interaction techniques for similar information structures.","PeriodicalId":198379,"journal":{"name":"Proceedings of the 2018 Conference on Human Information Interaction & Retrieval","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115843685","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Task-based Information Seeking in Different Study Settings","authors":"Yiwei Wang","doi":"10.1145/3176349.3176351","DOIUrl":"https://doi.org/10.1145/3176349.3176351","url":null,"abstract":"Existing studies have presented the relationships between task characteristics and individuals» information seeking and searching behaviors. Some task characteristics are found to have predictable influences on information seeking behaviors. However, most studies took place in lab settings and focused on individuals» interactions with information systems. How a laboratory environment affects individuals» natural information seeking behaviors is open to question. This paper proposes a study investigating the differences between information seeking behaviors in a lab setting where individuals» activities and resources are controlled and in naturalistic settings where individuals have access to all types of sources.","PeriodicalId":198379,"journal":{"name":"Proceedings of the 2018 Conference on Human Information Interaction & Retrieval","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125271138","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Fredrik Espinoza, Ola Hamfors, Jussi Karlgren, Fredrik Olsson, Per Persson, L. Hamberg, Magnus Sahlgren
{"title":"Analysis of Open Answers to Survey Questions through Interactive Clustering and Theme Extraction","authors":"Fredrik Espinoza, Ola Hamfors, Jussi Karlgren, Fredrik Olsson, Per Persson, L. Hamberg, Magnus Sahlgren","doi":"10.1145/3176349.3176892","DOIUrl":"https://doi.org/10.1145/3176349.3176892","url":null,"abstract":"This paper describes design principles for and the implementation of Gavagai Explorer---a new application which builds on interactive text clustering to extract themes from topically coherent text sets such as open text answers to surveys or questionnaires. An automated system is quick, consistent, and has full coverage over the study material. A system allows an analyst to analyze more answers in a given time period; provides the same initial results regardless of who does the analysis, reducing the risks of inter-rater discrepancy; and does not risk miss responses due to fatige or boredom. These factors reduce the cost and increase the reliability of the service. The most important feature, however, is relieving the human analyst from the frustrating aspects of the coding task, freeing the effort to the central challenge of understanding themes. Gavagai Explorer is available on-line.","PeriodicalId":198379,"journal":{"name":"Proceedings of the 2018 Conference on Human Information Interaction & Retrieval","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116845799","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"How do Information Source Selection Strategies Influence Users' Learning Outcomes'","authors":"Chang Liu, Xiaoxuan Song","doi":"10.1145/3176349.3176876","DOIUrl":"https://doi.org/10.1145/3176349.3176876","url":null,"abstract":"Learning-related type of tasks has attracted much research attention recently but it is still not clear what factors would influence users learning outcomes and how. In this study, we conducted a user experiment to assess searchers learning outcomes and examine how information source selection strategies would influence their learning outcomes. In this experiment, thirty-two college students conducted search for two types of learning tasks: receptive tasks and critical tasks. Participants were asked to write down what they knew about the task before and after the search. For data analysis, we proposed a comprehensive assessment method, which used both quantitative measures (i.e. knowledge points, knowledge facets, knowledge scope, etc.) and qualitative measures to assess users' learning outcomes. Our results demonstrated that searchers' information source preferences influence their learning outcomes; i.e., encyclopedia-preferred sessions had better relevance of written summaries in receptive tasks and Q&A preferred sessions led to better relevance in critical tasks. Furthermore, searchers had two types of information source selection strategies: task-adaptive strategy and non-task-adaptive strategy. The results showed that searchers with task-adaptive strategy could gain better learning outcomes, e.g. knowledge points, facets, scope, depth, relevance and analyticity. This study highlighted the importance of information source selection strategies in learning-related type of tasks, and knowing how to select suitable information sources for different types of tasks may benefit the learning outcome for searchers.","PeriodicalId":198379,"journal":{"name":"Proceedings of the 2018 Conference on Human Information Interaction & Retrieval","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129601855","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Collaborative Information Seeking through Social Media Updates in Real-Time","authors":"K. Bhat, Andrés Moreno, M. Best","doi":"10.1145/3176349.3176869","DOIUrl":"https://doi.org/10.1145/3176349.3176869","url":null,"abstract":"This paper describes co-located collaborative information seeking in the context of the Social Media Tracking Centre (SMTC) in Ghana. The SMTC was operational for three days during the Presidential elections in Dec 2016. The SMTC»s role was to collaboratively find and verify novel, actionable, and relevant information on social media and escalate it to the authorities to use that information to ensure a transparent and peaceful election process. We performed a qualitative analysis of semi-structured interviews with the volunteers at the SMTC as well as its managing team. We present, in this paper, the importance of volunteer motivation and co-location in the success of the SMTC, as well as the users» feedback on the collaborative tool, informing future design, derived from our analysis.","PeriodicalId":198379,"journal":{"name":"Proceedings of the 2018 Conference on Human Information Interaction & Retrieval","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115048553","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Towards Human-Like Conversational Search Systems","authors":"Mateusz Dubiel","doi":"10.1145/3176349.3176360","DOIUrl":"https://doi.org/10.1145/3176349.3176360","url":null,"abstract":"Voice search is currently widely available on the majority of mobile devices via use of Virtual Personal Assistants. However, despite its general availability, the use of voice interaction remains sporadic and is limited to basic search tasks such as checking weather updates and looking up answers to factual queries. Present-day voice search systems struggle to use relevant contextual information to maintain conversational state, and lack conversational initiative needed to clarify user's intent, which hampers their usability and prevents users from engaging in more complex interaction activities. This research investigates the potential of a hypothesised interactive information retrieval system with human-like conversational abilities. To this end, we propose a series of usability studies that involve a working prototype of a conversational system that uses real time speech synthesis. The proposed experiments seek to provide empirical evidence that enabling a voice search system with human-like conversational abilities can lead to increased likelihood of its adoption.","PeriodicalId":198379,"journal":{"name":"Proceedings of the 2018 Conference on Human Information Interaction & Retrieval","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126543878","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}