发布求助
文献互助 智能选刊 最新文献

Conference and Labs of the Evaluation Forum最新文献

筛选
英文 中文
Evaluating Temporal Persistence Using Replicability Measures 使用可复制性度量评估时间持久性
Conference and Labs of the Evaluation Forum Pub Date : 2023-08-21 DOI: 10.48550/arXiv.2308.10549
Jüri Keller, Timo Breuer, Philipp Schaer
{"title":"Evaluating Temporal Persistence Using Replicability Measures","authors":"Jüri Keller, Timo Breuer, Philipp Schaer","doi":"10.48550/arXiv.2308.10549","DOIUrl":"https://doi.org/10.48550/arXiv.2308.10549","url":null,"abstract":"In real-world Information Retrieval (IR) experiments, the Evaluation Environment (EE) is exposed to constant change. Documents are added, removed, or updated, and the information need and the search behavior of users is evolving. Simultaneously, IR systems are expected to retain a consistent quality. The LongEval Lab seeks to investigate the longitudinal persistence of IR systems, and in this work, we describe our participation. We submitted runs of five advanced retrieval systems, namely a Reciprocal Rank Fusion (RRF) approach, ColBERT, monoT5, Doc2Query, and E5, to both sub-tasks. Further, we cast the longitudinal evaluation as a replicability study to better understand the temporal change observed. As a result, we quantify the persistence of the submitted runs and see great potential in this evaluation method.","PeriodicalId":232729,"journal":{"name":"Conference and Labs of the Evaluation Forum","volume":"53 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130189102","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
ARC-NLP at PAN 2023: Transition-Focused Natural Language Inference for Writing Style Detection 基于过渡的自然语言推理写作风格检测
Conference and Labs of the Evaluation Forum Pub Date : 2023-07-27 DOI: 10.48550/arXiv.2307.14913
Izzet Emre Kucukkaya, Umitcan Sahin, Cagri Toraman
{"title":"ARC-NLP at PAN 2023: Transition-Focused Natural Language Inference for Writing Style Detection","authors":"Izzet Emre Kucukkaya, Umitcan Sahin, Cagri Toraman","doi":"10.48550/arXiv.2307.14913","DOIUrl":"https://doi.org/10.48550/arXiv.2307.14913","url":null,"abstract":"The task of multi-author writing style detection aims at finding any positions of writing style change in a given text document. We formulate the task as a natural language inference problem where two consecutive paragraphs are paired. Our approach focuses on transitions between paragraphs while truncating input tokens for the task. As backbone models, we employ different Transformer-based encoders with warmup phase during training. We submit the model version that outperforms baselines and other proposed model versions in our experiments. For the easy and medium setups, we submit transition-focused natural language inference based on DeBERTa with warmup training, and the same model without transition for the hard setup.","PeriodicalId":232729,"journal":{"name":"Conference and Labs of the Evaluation Forum","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-07-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124012824","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 2
ARC-NLP at PAN 2023: Hierarchical Long Text Classification for Trigger Detection PAN 2023上的ARC-NLP:触发检测的分层长文本分类
Conference and Labs of the Evaluation Forum Pub Date : 2023-07-27 DOI: 10.48550/arXiv.2307.14912
Umitcan Sahin, Izzet Emre Kucukkaya, Cagri Toraman
{"title":"ARC-NLP at PAN 2023: Hierarchical Long Text Classification for Trigger Detection","authors":"Umitcan Sahin, Izzet Emre Kucukkaya, Cagri Toraman","doi":"10.48550/arXiv.2307.14912","DOIUrl":"https://doi.org/10.48550/arXiv.2307.14912","url":null,"abstract":"Fanfiction, a popular form of creative writing set within established fictional universes, has gained a substantial online following. However, ensuring the well-being and safety of participants has become a critical concern in this community. The detection of triggering content, material that may cause emotional distress or trauma to readers, poses a significant challenge. In this paper, we describe our approach for the Trigger Detection shared task at PAN CLEF 2023, where we want to detect multiple triggering content in a given Fanfiction document. For this, we build a hierarchical model that uses recurrence over Transformer-based language models. In our approach, we first split long documents into smaller sized segments and use them to fine-tune a Transformer model. Then, we extract feature embeddings from the fine-tuned Transformer model, which are used as input in the training of multiple LSTM models for trigger detection in a multi-label setting. Our model achieves an F1-macro score of 0.372 and F1-micro score of 0.736 on the validation set, which are higher than the baseline results shared at PAN CLEF 2023.","PeriodicalId":232729,"journal":{"name":"Conference and Labs of the Evaluation Forum","volume":"29 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-07-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128150645","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
SuryaKiran at MEDIQA-Sum 2023: Leveraging LoRA for Clinical Dialogue Summarization SuryaKiran在MEDIQA-Sum 2023:利用LoRA进行临床对话总结
Conference and Labs of the Evaluation Forum Pub Date : 2023-07-11 DOI: 10.48550/arXiv.2307.05162
Kunal Suri, Prakhar Mishra, Saumajit Saha, Ashutosh Kumar Singh
{"title":"SuryaKiran at MEDIQA-Sum 2023: Leveraging LoRA for Clinical Dialogue Summarization","authors":"Kunal Suri, Prakhar Mishra, Saumajit Saha, Ashutosh Kumar Singh","doi":"10.48550/arXiv.2307.05162","DOIUrl":"https://doi.org/10.48550/arXiv.2307.05162","url":null,"abstract":"Finetuning Large Language Models helps improve the results for domain-specific use cases. End-to-end finetuning of large language models is time and resource intensive and has high storage requirements to store the finetuned version of the large language model. Parameter Efficient Fine Tuning (PEFT) methods address the time and resource challenges by keeping the large language model as a fixed base and add additional layers, which the PEFT methods finetune. This paper demonstrates the evaluation results for one such PEFT method Low Rank Adaptation (LoRA), for Clinical Dialogue Summarization. The evaluation results show that LoRA works at par with end-to-end finetuning for a large language model. The paper presents the evaluations done for solving both the Subtask A and B from ImageCLEFmedical {https://www.imageclef.org/2023/medical}","PeriodicalId":232729,"journal":{"name":"Conference and Labs of the Evaluation Forum","volume":"22 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-07-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134426739","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Overview of BioASQ 2023: The eleventh BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answering BioASQ 2023概述:大规模生物医学语义索引和问答的第十一届BioASQ挑战
Conference and Labs of the Evaluation Forum Pub Date : 2023-07-11 DOI: 10.48550/arXiv.2307.05131
A. Nentidis, Georgios Katsimpras, Anastasia Krithara, Salvador Lima-López, Eulàlia Farré-Maduell, Luis Gasco, Martin Krallinger, G. Paliouras
{"title":"Overview of BioASQ 2023: The eleventh BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answering","authors":"A. Nentidis, Georgios Katsimpras, Anastasia Krithara, Salvador Lima-López, Eulàlia Farré-Maduell, Luis Gasco, Martin Krallinger, G. Paliouras","doi":"10.48550/arXiv.2307.05131","DOIUrl":"https://doi.org/10.48550/arXiv.2307.05131","url":null,"abstract":"This is an overview of the eleventh edition of the BioASQ challenge in the context of the Conference and Labs of the Evaluation Forum (CLEF) 2023. BioASQ is a series of international challenges promoting advances in large-scale biomedical semantic indexing and question answering. This year, BioASQ consisted of new editions of the two established tasks b and Synergy, and a new task (MedProcNER) on semantic annotation of clinical content in Spanish with medical procedures, which have a critical role in medical practice. In this edition of BioASQ, 28 competing teams submitted the results of more than 150 distinct systems in total for the three different shared tasks of the challenge. Similarly to previous editions, most of the participating systems achieved competitive performance, suggesting the continuous advancement of the state-of-the-art in the field.","PeriodicalId":232729,"journal":{"name":"Conference and Labs of the Evaluation Forum","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-07-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132544011","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 4
Enhancing Biomedical Text Summarization and Question-Answering: On the Utility of Domain-Specific Pre-Training 增强生物医学文本摘要与问答:基于特定领域预训练的应用
Conference and Labs of the Evaluation Forum Pub Date : 2023-07-10 DOI: 10.48550/arXiv.2307.04412
Dima Galat, Marian-Andrei Rizoiu
{"title":"Enhancing Biomedical Text Summarization and Question-Answering: On the Utility of Domain-Specific Pre-Training","authors":"Dima Galat, Marian-Andrei Rizoiu","doi":"10.48550/arXiv.2307.04412","DOIUrl":"https://doi.org/10.48550/arXiv.2307.04412","url":null,"abstract":"Biomedical summarization requires large datasets to train for text generation. We show that while transfer learning offers a viable option for addressing this challenge, an in-domain pre-training does not always offer advantages in a BioASQ summarization task. We identify a suitable model architecture and use it to show a benefit of a general-domain pre-training followed by a task-specific fine-tuning in the context of a BioASQ summarization task, leading to a novel three-step fine-tuning approach that works with only a thousand in-domain examples. Our results indicate that a Large Language Model without domain-specific pre-training can have a significant edge in some domain-specific biomedical text generation tasks.","PeriodicalId":232729,"journal":{"name":"Conference and Labs of the Evaluation Forum","volume":"126 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-07-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130203819","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
DWReCO at CheckThat! 2023: Enhancing Subjectivity Detection through Style-based Data Sampling 在CheckThat上玩完!2023:通过基于风格的数据采样增强主观性检测
Conference and Labs of the Evaluation Forum Pub Date : 2023-07-07 DOI: 10.48550/arXiv.2307.03550
Ipek Baris Schlicht, Lynn Khellaf, Defne Altiok
{"title":"DWReCO at CheckThat! 2023: Enhancing Subjectivity Detection through Style-based Data Sampling","authors":"Ipek Baris Schlicht, Lynn Khellaf, Defne Altiok","doi":"10.48550/arXiv.2307.03550","DOIUrl":"https://doi.org/10.48550/arXiv.2307.03550","url":null,"abstract":"This paper describes our submission for the subjectivity detection task at the CheckThat! Lab. To tackle class imbalances in the task, we have generated additional training materials with GPT-3 models using prompts of different styles from a subjectivity checklist based on journalistic perspective. We used the extended training set to fine-tune language-specific transformer models. Our experiments in English, German and Turkish demonstrate that different subjective styles are effective across all languages. In addition, we observe that the style-based oversampling is better than paraphrasing in Turkish and English. Lastly, the GPT-3 models sometimes produce lacklustre results when generating style-based texts in non-English languages.","PeriodicalId":232729,"journal":{"name":"Conference and Labs of the Evaluation Forum","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-07-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126552865","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
UIT-Saviors at MEDVQA-GI 2023: Improving Multimodal Learning with Image Enhancement for Gastrointestinal Visual Question Answering MEDVQA-GI 2023:改善胃肠道视觉问答的多模态学习与图像增强
Conference and Labs of the Evaluation Forum Pub Date : 2023-07-06 DOI: 10.48550/arXiv.2307.02783
T. M. Thai, A. T. Vo, Hao K. Tieu, Linh Bui, T. Nguyen
{"title":"UIT-Saviors at MEDVQA-GI 2023: Improving Multimodal Learning with Image Enhancement for Gastrointestinal Visual Question Answering","authors":"T. M. Thai, A. T. Vo, Hao K. Tieu, Linh Bui, T. Nguyen","doi":"10.48550/arXiv.2307.02783","DOIUrl":"https://doi.org/10.48550/arXiv.2307.02783","url":null,"abstract":"In recent years, artificial intelligence has played an important role in medicine and disease diagnosis, with many applications to be mentioned, one of which is Medical Visual Question Answering (MedVQA). By combining computer vision and natural language processing, MedVQA systems can assist experts in extracting relevant information from medical image based on a given question and providing precise diagnostic answers. The ImageCLEFmed-MEDVQA-GI-2023 challenge carried out visual question answering task in the gastrointestinal domain, which includes gastroscopy and colonoscopy images. Our team approached Task 1 of the challenge by proposing a multimodal learning method with image enhancement to improve the VQA performance on gastrointestinal images. The multimodal architecture is set up with BERT encoder and different pre-trained vision models based on convolutional neural network (CNN) and Transformer architecture for features extraction from question and endoscopy image. The result of this study highlights the dominance of Transformer-based vision models over the CNNs and demonstrates the effectiveness of the image enhancement process, with six out of the eight vision models achieving better F1-Score. Our best method, which takes advantages of BERT+BEiT fusion and image enhancement, achieves up to 87.25% accuracy and 91.85% F1-Score on the development test set, while also producing good result on the private test set with accuracy of 82.01%.","PeriodicalId":232729,"journal":{"name":"Conference and Labs of the Evaluation Forum","volume":"16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-07-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122244039","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
Transfer Learning with Semi-Supervised Dataset Annotation for Birdcall Classification 基于半监督数据集标注的迁移学习鸟类叫声分类
Conference and Labs of the Evaluation Forum Pub Date : 2023-06-29 DOI: 10.48550/arXiv.2306.16760
Anthony Miyaguchi, Nathan Zhong, Murilo Gustineli, Christopher Hayduk
{"title":"Transfer Learning with Semi-Supervised Dataset Annotation for Birdcall Classification","authors":"Anthony Miyaguchi, Nathan Zhong, Murilo Gustineli, Christopher Hayduk","doi":"10.48550/arXiv.2306.16760","DOIUrl":"https://doi.org/10.48550/arXiv.2306.16760","url":null,"abstract":"We present working notes on transfer learning with semi-supervised dataset annotation for the BirdCLEF 2023 competition, focused on identifying African bird species in recorded soundscapes. Our approach utilizes existing off-the-shelf models, BirdNET and MixIT, to address representation and labeling challenges in the competition. We explore the embedding space learned by BirdNET and propose a process to derive an annotated dataset for supervised learning. Our experiments involve various models and feature engineering approaches to maximize performance on the competition leaderboard. The results demonstrate the effectiveness of our approach in classifying bird species and highlight the potential of transfer learning and semi-supervised dataset annotation in similar tasks.","PeriodicalId":232729,"journal":{"name":"Conference and Labs of the Evaluation Forum","volume":"69 4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131954269","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Retrieving Comparative Arguments using Ensemble Methods and BERT 使用集成方法和BERT检索比较参数
Conference and Labs of the Evaluation Forum Pub Date : 2023-05-01 DOI: 10.48550/arXiv.2305.01513
V. Chekalina, A. Panchenko
{"title":"Retrieving Comparative Arguments using Ensemble Methods and BERT","authors":"V. Chekalina, A. Panchenko","doi":"10.48550/arXiv.2305.01513","DOIUrl":"https://doi.org/10.48550/arXiv.2305.01513","url":null,"abstract":"In this paper, we present a submission to the Touche lab's Task 2 on Argument Retrieval for Comparative Questions. Our team Katana supplies several approaches based on decision tree ensembles algorithms to rank comparative documents in accordance with their relevance and argumentative support. We use PyTerrier library to apply ensembles models to a ranking problem, considering statistical text features and features based on comparative structures. We also employ large contextualized language modelling techniques, such as BERT, to solve the proposed ranking task. To merge this technique with ranking modelling, we leverage neural ranking library OpenNIR. Our systems substantially outperforming the proposed baseline and scored first in relevance and second in quality according to the official metrics of the competition (for measure NDCG@5 score). Presented models could help to improve the performance of processing comparative queries in information retrieval and dialogue systems.","PeriodicalId":232729,"journal":{"name":"Conference and Labs of the Evaluation Forum","volume":"29 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124045516","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 3
下一页 尾页
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整，为了账户安全，请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
请完成安全验证×
相关产品
×
本文献相关产品
联系我们：info@booksci.cn Book学术提供免费学术资源搜索服务，方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号：481959085
Book学术官方微信