Proceedings of the 4th and 5th Annual Meetings of the Forum for Information Retrieval Evaluation最新文献

{"title":"Named-Entity Recognition in Bengali","authors":"Apurbalal Senapati, Arjun Das, Utpal Garain","doi":"10.1145/2701336.2701647","DOIUrl":"https://doi.org/10.1145/2701336.2701647","url":null,"abstract":"This paper describes two systems for Named Entity Recognition (NER) and performance of two systems has been compared. The first system is a rule-based one whereas the second one is statistical (based on CRF) in nature. The systems vary in some other aspects too, for example, the first system works on untagged data (not even POS tag is done) to identify NER whereas the second system makes use of a POS tagger and a chunker. The rules used by the first system are mined from the training data. The CRF-based classification does not require any explicit linguistic rules but it uses a gazetteer built from Wiki and other sources.","PeriodicalId":405191,"journal":{"name":"Proceedings of the 4th and 5th Annual Meetings of the Forum for Information Retrieval Evaluation","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129838769","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Pre-Retrieval based Strategies for Cross Language News Story Search","authors":"Aarti Kumar, S. Das","doi":"10.1145/2701336.2701640","DOIUrl":"https://doi.org/10.1145/2701336.2701640","url":null,"abstract":"The task of measuring text reuse and linking target and source documents becomes challenging if reused text has been translated in other language and even more challenging if translation language uses a different script and syntactical structure. This year, in CLINSS, focus was on journalistic text reuse between texts written in two different languages. This paper focuses on identifying text reuse between texts written in two different languages-English and Hindi and aims at evaluating the CLINSS dataset by linking 25 target English news stories with top 100 Hindi news stories out of corpora of 50691 stories. This is our first participation at CLINSS task and we have submitted two sets of runs as MANIT-1 and MANIT-2. In MANIT-1 two pre-retrieval strategies 1) Query formed using Proper Noun and 2) Query formed using words whose frequency is equal to or higher than average frequency are used to formulate the query. Query is translated using either dictionary based or machine translation CLIR based approach before retrieving the documents. In Run 1 and 2 pre-retrieval strategies clubbed up with CLIR's dictionary based approach is used to link English news stories with Hindi news stories. In Run-3 instead of using dictionary based approach, freely available online Google translate [18] and online Changathi Hindi transliterater [6] is used as Hindi resources for translating/transliterating the query. Through MANIT-1 runs the attempt is to evaluate the performance of two pre-retrieval strategies and to compare the existing dictionary based and machine translation based approaches of CLIR. It is observed that dictionary based approach clubbed up with proper noun based pre-retrieval strategy performed better.","PeriodicalId":405191,"journal":{"name":"Proceedings of the 4th and 5th Annual Meetings of the Forum for Information Retrieval Evaluation","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115436710","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Argumentation Mining: Where are we now, where do we want to be and how do we get there?","authors":"Marie-Francine Moens","doi":"10.1145/2701336.2701635","DOIUrl":"https://doi.org/10.1145/2701336.2701635","url":null,"abstract":"This paper gives a short overview of the state-of-the-art and goals of argumentation mining and it provides ideas for further research. Its content is based on two invited lectures on argumentation mining respectively at the FIRE 2013 conference at the India International Center in New Delhi, India and a lecture given as SICSA distinguished visitor at the University of Dundee, UK in the summer of 2014.","PeriodicalId":405191,"journal":{"name":"Proceedings of the 4th and 5th Annual Meetings of the Forum for Information Retrieval Evaluation","volume":"36 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125363849","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"ISM@FIRE-2013 Information Access in The Legal Domain","authors":"Ambedkar Kanapala, Sukomal Pal","doi":"10.1145/2701336.2701645","DOIUrl":"https://doi.org/10.1145/2701336.2701645","url":null,"abstract":"This paper describes the work that we did as a participant in the task of Adhoc retrieval from legal documents (Consumer Law and Hindu Marriage & Divorce Law) at FIRE 2013. We managed two runs using the Indri search retrieval system as official submission. However, post-submission, we experimented with ten different models using the TERRIER system. With the available qrels, we could evaluate only the runs pertaining to Consumer Law. Initial results have shown substantial improvement in retrieval performance. Our experimentation has also resulted in preparing a cookbook for optimal parameter-setting for a number of Terrier models. In Identification and Classification of Propositions in Court Judgment (parse each judgment into individual propositions), we have written a program which takes a single file as an input from the corpus and breaks the data of the given file into individual sentences. In summary, ours was a rich learning experience which needs to be continued to gain further insights. Also, we believe as pilot track, information access from legal domain is promising enough to be continued further as it has potential of being a fertile research area, specifically in Indian context.","PeriodicalId":405191,"journal":{"name":"Proceedings of the 4th and 5th Annual Meetings of the Forum for Information Retrieval Evaluation","volume":"13 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125267853","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Applying Query Formulation and Fusion Techniques For Cross Language News Story Search","authors":"Piyush Arora, Jennifer Foster, G. Jones","doi":"10.1145/2701336.2701650","DOIUrl":"https://doi.org/10.1145/2701336.2701650","url":null,"abstract":"Cross Language News story search (CLNSS) is concerned with finding documents describing the same events in documents in different languages. As well as supporting information retrieval (IR), CLNSS has other applications in mining parallel and comparable data across different languages. In this paper, we present an overview of the work carried out for our participation in the Cross Language !ndian News Story Search (CL!NSS) task at FIRE 2013. In the CL!NSS task we explored the problem of cross language news search for the English-Hindi language pair. English news stories are used as queries to seek similar news documents from Hindi news articles. Hindi being a resource-scarce language offers many challenges towards retrieving relevant news articles. We investigate and contrast translation of input queries from English to Hindi using the Google and Bing translation services. To support translation of out-of-vocabulary words we use the Google transliteration service. A key challenge of the CL!NSS task is formation of search queries from the English news articles, since they are much longer than the much shorter queries typically used in IR applications. To address this problem, we explore the use of summarization to extract a query from the input news documents, and use these summarized queries as the input to the cross language IR system. We explore the use of query expansion using pseudo relevance feedback (PRF) in the IR process, since this has been shown to be effective for cross language IR in many previous investigations. We also explore in detail the use of data fusion techniques over different sets of retrieved results obtained using diverse query formulation techniques. For the CL!NSS task our team submitted 3 main runs. The results of our best run was ranked first among official submissions based on NDCG@5 and NDCG@10 values and second for NDCG@1 values. For the 25 test queries the results of our best main run were NDCG@1 0.7400, NDCG@5 0.6809 and NDCG@10 0.7268. We present our methodology, official results and results of a number of post-task experiments that were conducted to further examine the cross language search problem. Our experiments reveal that query formulation plays a vital role in improving search results for news documents across different languages. Instead of using the complete news documents the summarized queries show better performance. Data fusion techniques also help to improve the performance of the system by boosting the rank of documents, thus improving the NDCG scores.","PeriodicalId":405191,"journal":{"name":"Proceedings of the 4th and 5th Annual Meetings of the Forum for Information Retrieval Evaluation","volume":"67 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122819932","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A System for Classification of Propositions of the Indian Supreme Court Judgements","authors":"Nitin Ramrakhiyani, Sachin Pawar, Girish Keshav Palshikar","doi":"10.1145/2701336.2701643","DOIUrl":"https://doi.org/10.1145/2701336.2701643","url":null,"abstract":"In this work, we describe a system for classification of propositions from legal judgements of the Supreme Court of India. The system was submitted for participation to the Information Access in the Legal Domain track at the Forum for Information Retrieval Evaluation (FIRE) 2013. The system uses a multi-class Maximum Entropy classifier and various specially designed features to capture the underlying characteristics of the legal propositions. The best performing feature set was chosen by 10-fold cross-validation over the training set. The system achieved an accuracy of 65.03% on the training set and an accuracy of 51.02% on the test set.","PeriodicalId":405191,"journal":{"name":"Proceedings of the 4th and 5th Annual Meetings of the Forum for Information Retrieval Evaluation","volume":"258 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124239160","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Proceedings of the 4th and 5th Annual Meetings of the Forum for Information Retrieval Evaluation","authors":"Prasenjit Majumder, Mandar Mitra, Madhulika Agrawal, Parth Mehta","doi":"10.1145/2701336","DOIUrl":"https://doi.org/10.1145/2701336","url":null,"abstract":"Forum for Information Retrieval Evaluation (FIRE) is presenting the post proceedings for 2012 and 2013. FIRE '12 and FIRE '13 was held at Indian Statistical Institute, Kolkata, India and India International Centre, New Delhi, India respectively. FIRE focuses on providing a platform for the community to work in collaboration for developing and evaluating information access technologies in South Asian languages by providing large scale test collection.","PeriodicalId":405191,"journal":{"name":"Proceedings of the 4th and 5th Annual Meetings of the Forum for Information Retrieval Evaluation","volume":"20 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115311113","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The FIRE 2013 Question Answering for the Spoken Web Task","authors":"Douglas W. Oard, Jerome White, Jiaul H. Paik, Rashmi Sankepally, A. Jansen","doi":"10.1145/2701336.2701638","DOIUrl":"https://doi.org/10.1145/2701336.2701638","url":null,"abstract":"The FIRE 2013 Question Answering for the Spoken Web (QASW) task was an information retrieval evaluation in which the goal was to match spoken Gujarati questions to spoken Gujarati answers. This paper describes the design of the task, the development of the test collection, the runs that were submitted, and the corresponding results.","PeriodicalId":405191,"journal":{"name":"Proceedings of the 4th and 5th Annual Meetings of the Forum for Information Retrieval Evaluation","volume":"6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126544896","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Overview of the FIRE 2013 Track on Transliterated Search","authors":"Rishiraj Saha Roy, M. Choudhury, Prasenjit Majumder, Komal Agarwal","doi":"10.1145/2701336.2701636","DOIUrl":"https://doi.org/10.1145/2701336.2701636","url":null,"abstract":"In this paper, we provide an overview of the FIRE 2013 track on transliterated search and describe the datasets released as part of the track. This was the first year that the track was organized. We had proposed two subtasks as part of the challenge. In the first subtask, which we had proposed for Hindi, Bangla, and Gujarati, participants had to devise an algorithm to label the true languages of words in a sentence. Additionally, if a non-English word was identified, the algorithm was also supposed to provide the transliteration of the word in the native script. The second subtask was retrieval-based, where mixed-script documents had to be retrieved and ranked by relevance in response to ad hoc queries. The queries in our dataset were Bollywood Hindi song lyrics, in Roman script. We received a total of 25 run submissions from five different teams across the world (three from India and two from abroad). Conducting this track helped us generate awareness about the importance of transliteration in the context of Indian languages. Results show that there is considerable scope for improvement of transliteration accuracies for the studied languages.","PeriodicalId":405191,"journal":{"name":"Proceedings of the 4th and 5th Annual Meetings of the Forum for Information Retrieval Evaluation","volume":"107 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124957993","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Weighted Edit Distance based FAQ Retrieval using Noisy Queries","authors":"Shahbaaz Mhaisale, Sangameshwar Patil, K. Mahamuni","doi":"10.1145/2701336.2701644","DOIUrl":"https://doi.org/10.1145/2701336.2701644","url":null,"abstract":"In this paper, we describe our contribution to the FIRE 2013 shared task on \"FAQ Retrieval using Noisy Queries\". Short messaging service (SMS) and voice-based interfaces such as Siri have become quite popular for quick information retrieval these days. The problem posed in this FIRE 2013 shared task is that given an SMS query or speech-to-text transcript of a voice based query, find the best matching frequently asked question (FAQ) from a database or return NULL if there is no relevant FAQ. In our solution, we first normalize the query and then compute the similarity between the query and each question using weighted edit distance. We observe that instead of using the standard edit distance which gives equal weight for all the string edit operations (such as insertion, deletion, transposition, substitution), weighted edit distance works significantly better for checking similarity of SMS queries with FAQ database questions. We determine whether a query is out of domain (i.e. cannot be answered using current set of questions in the FAQ database) using a simple classifier using FAQ database itself. Our method identified 105 out of 148 out-of-domain queries correctly and 188 out of 392 in-domain queries correctly with mean reciprocal rank (MRR) of 0.8836. Since we observed very little correlation in the transcript of speech queries with best matching question in the training data, we also report the results for SMS only queries. On the SMS queries, our method identified 188 of 200 queries correctly, resulting in an MRR of 0.968.","PeriodicalId":405191,"journal":{"name":"Proceedings of the 4th and 5th Annual Meetings of the Forum for Information Retrieval Evaluation","volume":"49 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129273267","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}