发布求助
文献互助 智能选刊 最新文献

LivingLab '13最新文献

筛选
英文 中文
A month in the life of a production news recommender system 一个月的生产新闻推荐系统的生命
LivingLab '13 Pub Date : 2013-11-01 DOI: 10.1145/2513150.2513159
A. Said, Jimmy J. Lin, Alejandro Bellogín, A. D. Vries
{"title":"A month in the life of a production news recommender system","authors":"A. Said, Jimmy J. Lin, Alejandro Bellogín, A. D. Vries","doi":"10.1145/2513150.2513159","DOIUrl":"https://doi.org/10.1145/2513150.2513159","url":null,"abstract":"During the last decade, recommender systems have become a ubiquitous feature in the online world. Research on systems and algorithms in this area has flourished, leading to novel techniques for personalization and recommendation. The evaluation of recommender systems, however, has not seen similar progress---techniques have changed little since the advent of recommender systems, when evaluation methodologies were \"borrowed\" from related research areas. As an effort to move evaluation methodology forward, this paper describes a production recommender system infrastructure that allows research systems to be evaluated in situ, by real-world metrics such as user clickthrough. We present an analysis of one month of interactions with this infrastructure and share our findings.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"10 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128398206","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 27
Evaluation for operational IR applications: generalizability and automation 可操作IR应用的评估:通用性和自动化
LivingLab '13 Pub Date : 2013-11-01 DOI: 10.1145/2513150.2513160
Melanie Imhof, Martin Braschler, P. Hansen, Stefan Rietberger
{"title":"Evaluation for operational IR applications: generalizability and automation","authors":"Melanie Imhof, Martin Braschler, P. Hansen, Stefan Rietberger","doi":"10.1145/2513150.2513160","DOIUrl":"https://doi.org/10.1145/2513150.2513160","url":null,"abstract":"Black box information retrieval (IR) application evaluation allows practitioners to measure the quality of their IR application. Instead of evaluating specific components, e.g. solely the search engine, a complete IR application, including the user's perspective, is evaluated. The evaluation methodology is designed to be applicable to operational IR applications. The black box evaluation methodology could be packaged into an evaluation and monitoring tool, making it usable for industry stakeholders. The tool should lead practitioners through the evaluation process and maintain the test results for the manual and automatic tests. This paper shows that the methodology is generalizable, even though the diversity of IR applications is high. The challenges in automating tests are the simulation of tasks that require intellectual effort and the handling of different visualizations of the same concept.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"68 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124915751","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Using CrowdLogger for in situ information retrieval system evaluation 利用CrowdLogger对现场信息检索系统进行评价
LivingLab '13 Pub Date : 2013-11-01 DOI: 10.1145/2513150.2513164
H. Feild, James Allan
{"title":"Using CrowdLogger for in situ information retrieval system evaluation","authors":"H. Feild, James Allan","doi":"10.1145/2513150.2513164","DOIUrl":"https://doi.org/10.1145/2513150.2513164","url":null,"abstract":"A major hurdle faced by many information retrieval researchers---especially in academia---is evaluating retrieval systems in the wild. Challenges include tapping into large user bases, collecting user behavior, and modifying a given retrieval system. We outline several options available to researchers to overcome these challenges along with their advantages and disadvantages. We then demonstrate how CrowdLogger, an open-source browser extension for Firefox and Google Chrome, can be used as an in situ evaluation platform.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"2000 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121074343","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 4
Online metrics for web search relevance 网络搜索相关性的在线指标
LivingLab '13 Pub Date : 2013-11-01 DOI: 10.1145/2513150.2513165
Jan O. Pedersen
{"title":"Online metrics for web search relevance","authors":"Jan O. Pedersen","doi":"10.1145/2513150.2513165","DOIUrl":"https://doi.org/10.1145/2513150.2513165","url":null,"abstract":"Information Retrieval has a long tradition of being metrics driven. Ranking algorithms are assessed with respect to some utility measure that reflects the likelihood of satisfying an information need. Traditionally these metrics are based on offline judgments. This is very flexible since judgments can be made for any desired output. However, judgments are no better than judgment guidelines and are at some distance from the actual user experience. Modern Web Search engines enjoy an additional resource; existing web search traffic and its attendant wealth of user engagement data. Primarily this signal consists of logged queries and user actions, including clicks and reformulations. I will discuss how this data can be used to derive Web Search quality metrics that have very different properties than traditional offline metrics.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123606145","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
A private living lab for requirements based evaluation 一个基于需求评估的私人生活实验室
LivingLab '13 Pub Date : 2013-11-01 DOI: 10.1145/2513150.2513158
Christian Beutenmüller, Stefan Bordag, Ramin Assadollahi
{"title":"A private living lab for requirements based evaluation","authors":"Christian Beutenmüller, Stefan Bordag, Ramin Assadollahi","doi":"10.1145/2513150.2513158","DOIUrl":"https://doi.org/10.1145/2513150.2513158","url":null,"abstract":"A \"Living Lab\" is described as an open innovation space for the cooperation of users, researchers and even companies to participate in a common process to develop innovative solutions. An architecture for a living lab for IR has been proposed in [1]. In this paper we propose a method and system that foregoes the inherent openness of the living lab and implements a private living lab to enable the cooperation between a research department, agile software development, and requirements engineering/quality assurance.\u0000 This allows the research department to overcome the limitations of the usual approach to gold standard-based evaluation, while preserving its positive aspects. The definition of a private living lab may be seen along the lines of the separation of public and private clouds in cloud computing.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132753030","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
Factors affecting conditions of trust in participant recruiting and retention: a position paper 影响参与者招募和保留信任条件的因素:立场文件
LivingLab '13 Pub Date : 2013-11-01 DOI: 10.1145/2513150.2513161
Catherine L. Smith
{"title":"Factors affecting conditions of trust in participant recruiting and retention: a position paper","authors":"Catherine L. Smith","doi":"10.1145/2513150.2513161","DOIUrl":"https://doi.org/10.1145/2513150.2513161","url":null,"abstract":"This paper contemplates some of the challenges faced in recruiting and developing a community of contributors (participants/subjects) for a living laboratory for IR evaluation. We briefly review several factors that may affect the efficacy of participant recruiting. The potential benefits of collaboration with respect to recruiting are also discussed briefly.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"46 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127231868","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 3
Lerot: an online learning to rank framework Lerot:一个在线学习排名框架
LivingLab '13 Pub Date : 2013-11-01 DOI: 10.1145/2513150.2513162
Anne Schuth, Katja Hofmann, Shimon Whiteson, M. de Rijke
{"title":"Lerot: an online learning to rank framework","authors":"Anne Schuth, Katja Hofmann, Shimon Whiteson, M. de Rijke","doi":"10.1145/2513150.2513162","DOIUrl":"https://doi.org/10.1145/2513150.2513162","url":null,"abstract":"Online learning to rank methods for IR allow retrieval systems to optimize their own performance directly from interactions with users via click feedback. In the software package Lerot, presented in this paper, we have bundled all ingredients needed for experimenting with online learning to rank for IR. Lerot includes several online learning algorithms, interleaving methods and a full suite of ways to evaluate these methods. In the absence of real users, the evaluation method bundled in the software package is based on simulations of users interacting with the search engine. The software presented here has been used to verify findings of over six papers at major information retrieval venues over the last few years.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"35 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123635767","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 49
FindiLike: a preference driven entity search engine for evaluating entity retrieval and opinion summarization FindiLike:一个偏好驱动的实体搜索引擎，用于评估实体检索和意见总结
LivingLab '13 Pub Date : 2013-11-01 DOI: 10.1145/2513150.2513163
Kavita A. Ganesan, ChengXiang Zhai
{"title":"FindiLike: a preference driven entity search engine for evaluating entity retrieval and opinion summarization","authors":"Kavita A. Ganesan, ChengXiang Zhai","doi":"10.1145/2513150.2513163","DOIUrl":"https://doi.org/10.1145/2513150.2513163","url":null,"abstract":"We describe a novel preference-driven search engine (FindiLike) which allows users to find entities of interest based on preferences and also allows users to digest opinions about the retrieved entities easily. FindiLike leverages large amounts of online reviews about various entities, and ranks entities based on how well their associated reviews match a user's preference query (expressed in keywords). FindiLike then uses abstractive summarization techniques to generate concise opinion summaries to enable users to digest the opinions about an entity. We discuss how the system can be extended to support in situ evaluation of two interesting new tasks, i.e., opinion-based entity ranking and abstractive summarization of opinions. The system is currently supporting hotel search and being extended to support in situ evaluation of these two tasks. We will demonstrate the system in the domain of hotel search and show how in situ evaluation can be supported through natural user interaction with the system.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"149 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132213291","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整，为了账户安全，请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
请完成安全验证×
相关产品
×
本文献相关产品
联系我们：info@booksci.cn Book学术提供免费学术资源搜索服务，方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号：481959085
Book学术官方微信