AI Open最新文献_第10页

WuDaoCorpora: A super large-scale Chinese corpora for pre-training language models 五道语料库:一个用于预训练语言模型的超大规模汉语语料库

AI Open Pub Date : 2021-01-01 DOI: 10.1016/j.aiopen.2021.06.001

Sha Yuan , Hanyu Zhao , Zhengxiao Du , Ming Ding , Xiao Liu , Yukuo Cen , Xu Zou , Zhilin Yang , Jie Tang

{"title":"WuDaoCorpora: A super large-scale Chinese corpora for pre-training language models","authors":"Sha Yuan , Hanyu Zhao , Zhengxiao Du , Ming Ding , Xiao Liu , Yukuo Cen , Xu Zou , Zhilin Yang , Jie Tang","doi":"10.1016/j.aiopen.2021.06.001","DOIUrl":"10.1016/j.aiopen.2021.06.001","url":null,"abstract":"<div><p>Using large-scale training data to build a pre-trained language model (PLM) with a larger volume of parameters can significantly improve downstream tasks. For example, OpenAI trained the GPT3 model with 175 billion parameters on 570 GB English training data, enabling downstream applications building with only a small number of samples. However, there is a lack of Chinese corpus to support large-scale PLMs. This paper introduces a super large-scale Chinese corpora WuDaoCorpora, containing about 3 TB training data and 1.08 trillion Chinese characters. We also release the base version of WuDaoCorpora, containing about 200 GB training data and 72 billion Chinese characters. As a baseline, we train a model transformer-XL with 3 billion parameters on the base version to test the corpora's effect. The results show that the models trained on this corpora can achieve excellent performance in Chinese. The data and model are available at <span>https://data.wudaoai.cn</span><svg><path></path></svg> and <span>https://github.com/THUDM/Chinese-Transformer-XL</span><svg><path></path></svg>, respectively.</p></div>","PeriodicalId":100068,"journal":{"name":"AI Open","volume":"2 ","pages":"Pages 65-68"},"PeriodicalIF":0.0,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1016/j.aiopen.2021.06.001","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"81139879","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 54

Know what you don't need: Single-Shot Meta-Pruning for attention heads 知道什么是你不需要的:一次性修剪注意力

AI Open Pub Date : 2021-01-01 DOI: 10.1016/j.aiopen.2021.05.003

Zhengyan Zhang , Fanchao Qi , Zhiyuan Liu , Qun Liu , Maosong Sun

{"title":"Know what you don't need: Single-Shot Meta-Pruning for attention heads","authors":"Zhengyan Zhang , Fanchao Qi , Zhiyuan Liu , Qun Liu , Maosong Sun","doi":"10.1016/j.aiopen.2021.05.003","DOIUrl":"10.1016/j.aiopen.2021.05.003","url":null,"abstract":"<div><p>Deep pre-trained Transformer models have achieved state-of-the-art results over a variety of natural language processing (NLP) tasks. By learning rich language knowledge with millions of parameters, these models are usually overparameterized and significantly increase the computational overhead in applications. It is intuitive to address this issue by model compression. In this work, we propose a method, called Single-Shot Meta-Pruning, to compress deep pre-trained Transformers before fine-tuning. Specifically, we focus on pruning unnecessary attention heads adaptively for different downstream tasks. To measure the informativeness of attention heads, we train our Single-Shot Meta-Pruner (SMP) with a meta-learning paradigm aiming to maintain the distribution of text representations after pruning. Compared with existing compression methods for pre-trained models, our method can reduce the overhead of both fine-tuning and inference. Experimental results show that our pruner can selectively prune 50% of attention heads with little impact on the performance on downstream tasks and even provide better text representations. The source code is available at <span>https://github.com/thunlp/SMP</span><svg><path></path></svg>.</p></div>","PeriodicalId":100068,"journal":{"name":"AI Open","volume":"2 ","pages":"Pages 36-42"},"PeriodicalIF":0.0,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1016/j.aiopen.2021.05.003","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"75769316","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 19

Rule-based data augmentation for knowledge graph embedding 基于规则的知识图嵌入数据增强

AI Open Pub Date : 2021-01-01 DOI: 10.1016/j.aiopen.2021.09.003

Guangyao Li , Zequn Sun , Lei Qian , Qiang Guo , Wei Hu

{"title":"Rule-based data augmentation for knowledge graph embedding","authors":"Guangyao Li , Zequn Sun , Lei Qian , Qiang Guo , Wei Hu","doi":"10.1016/j.aiopen.2021.09.003","DOIUrl":"10.1016/j.aiopen.2021.09.003","url":null,"abstract":"<div><p>Knowledge graph (KG) embedding models suffer from the incompleteness issue of observed facts. Different from existing solutions that incorporate additional information or employ expressive and complex embedding techniques, we propose to augment KGs by iteratively mining logical rules from the observed facts and then using the rules to generate new relational triples. We incrementally train KG embeddings with the coming of new augmented triples, and leverage the embeddings to validate these new triples. To guarantee the quality of the augmented data, we filter out the noisy triples based on a propagation mechanism during the validation. The mined rules and rule groundings are human-understandable, and can make the augmentation procedure reliable. Our KG augmentation framework is applicable to any KG embedding models with no need to modify their embedding techniques. Our experiments on two popular embedding-based tasks (i.e., entity alignment and link prediction) show that the proposed framework can bring significant improvement to existing KG embedding models on most benchmark datasets.</p></div>","PeriodicalId":100068,"journal":{"name":"AI Open","volume":"2 ","pages":"Pages 186-196"},"PeriodicalIF":0.0,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2666651021000267/pdfft?md5=46899106b76601dcb62a0d1c184db35c&pid=1-s2.0-S2666651021000267-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"87043858","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 6

AI-driven drug discovery: A boon against COVID-19? 人工智能驱动的药物发现:对抗COVID-19的福音?

AI Open Pub Date : 2020-01-01 DOI: 10.1016/j.aiopen.2020.07.001

Aman Chandra Kaushik , Utkarsh Raj

引用次数: 22

Extracting Events and Their Relations from Texts: A Survey on Recent Research Progress and Challenges 从文本中提取事件及其关系:近期研究进展与挑战综述

AI Open Pub Date : 2020-01-01 DOI: 10.1016/j.aiopen.2021.02.004

Kang Liu , Yubo Chen , Jian Liu , Xinyu Zuo , Jun Zhao

引用次数: 22

Graph neural networks: A review of methods and applications 图神经网络:方法和应用综述

AI Open Pub Date : 2020-01-01 DOI: 10.1016/j.aiopen.2021.01.001

Jie Zhou , Ganqu Cui , Shengding Hu , Zhengyan Zhang , Cheng Yang , Zhiyuan Liu , Lifeng Wang , Changcheng Li , Maosong Sun

{"title":"Graph neural networks: A review of methods and applications","authors":"Jie Zhou , Ganqu Cui , Shengding Hu , Zhengyan Zhang , Cheng Yang , Zhiyuan Liu , Lifeng Wang , Changcheng Li , Maosong Sun","doi":"10.1016/j.aiopen.2021.01.001","DOIUrl":"10.1016/j.aiopen.2021.01.001","url":null,"abstract":"<div><p>Lots of learning tasks require dealing with graph data which contains rich relation information among elements. Modeling physics systems, learning molecular fingerprints, predicting protein interface, and classifying diseases demand a model to learn from graph inputs. In other domains such as learning from non-structural data like texts and images, reasoning on extracted structures (like the dependency trees of sentences and the scene graphs of images) is an important research topic which also needs graph reasoning models. Graph neural networks (GNNs) are neural models that capture the dependence of graphs via message passing between the nodes of graphs. In recent years, variants of GNNs such as graph convolutional network (GCN), graph attention network (GAT), graph recurrent network (GRN) have demonstrated ground-breaking performances on many deep learning tasks. In this survey, we propose a general design pipeline for GNN models and discuss the variants of each component, systematically categorize the applications, and propose four open problems for future research.</p></div>","PeriodicalId":100068,"journal":{"name":"AI Open","volume":"1 ","pages":"Pages 57-81"},"PeriodicalIF":0.0,"publicationDate":"2020-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1016/j.aiopen.2021.01.001","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"88004694","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 3407

AI OPEN inaugural editorial AI OPEN首期社论

AI Open Pub Date : 2020-01-01 DOI: 10.1016/j.aiopen.2021.04.001

Jie Tang (Professor and Associate Chair)

引用次数: 0

Neural machine translation: A review of methods, resources, and tools 神经机器翻译:方法、资源和工具的回顾

AI Open Pub Date : 2020-01-01 DOI: 10.1016/j.aiopen.2020.11.001

Zhixing Tan , Shuo Wang , Zonghan Yang , Gang Chen , Xuancheng Huang , Maosong Sun , Yang Liu

引用次数: 60

User behavior modeling for Web search evaluation 网络搜索评价的用户行为建模

AI Open Pub Date : 2020-01-01 DOI: 10.1016/j.aiopen.2021.02.003

Fan Zhang , Yiqun Liu , Jiaxin Mao , Min Zhang , Shaoping Ma