Proceedings of the 1st International Workshop on Trustworthy AI for Multimedia Computing最新文献

Hierarchical Semantic Enhanced Directional Graph Network for Visual Commonsense Reasoning 面向视觉常识推理的层次语义增强方向图网络

Proceedings of the 1st International Workshop on Trustworthy AI for Multimedia Computing Pub Date : 2021-10-22 DOI: 10.1145/3475731.3484957

Mingyan Wu, Shuhan Qi, Jun Rao, Jia-jia Zhang, Qing Liao, Xuan Wang, Xinxin Liao

{"title":"Hierarchical Semantic Enhanced Directional Graph Network for Visual Commonsense Reasoning","authors":"Mingyan Wu, Shuhan Qi, Jun Rao, Jia-jia Zhang, Qing Liao, Xuan Wang, Xinxin Liao","doi":"10.1145/3475731.3484957","DOIUrl":"https://doi.org/10.1145/3475731.3484957","url":null,"abstract":"Visual commonsense reasoning (VCR) task aims at boosting research of cognition-level correlations reasoning. It requires not only a thorough understanding of correlated details of the scene but also the ability to infer correlation with related commonsense knowledge. Existing approaches consider the region-word affinity to perform the semantic alignment between vision and linguistic domains, which neglect the implicit correspondence (e.g. word-scene, region-phrase, and phrase-scene) among visual concepts and linguistic words. Although efforts have been made to deliver promising results in previous work, these methods are still confronted with challenges when comes to make interpretable reasoning. Toward this end, we present a novel hierarchical semantic enhanced directional graph network. To be more specific, we design a Modality Interaction Unit (MIU) module, which captures high-order cross-modal alignment by aggregating the hierarchical vision-language relationships. Afterward, we propose a direction clue-aware graph reasoning (DCGR) module. In this module, valuable entities can be dynamically selected in each reasoning step, according to the importance of these entities. This leads to a more interpretable reasoning procedure. Ultimately, heterogeneous graph attention is introduced to filter the irrelevant parts of the final answers. Extensive experiments have been conducted on the VCR benchmark dataset, which demonstrates that our method can achieve competitive results and better interpretability compared with several state-of-the-art baselines.","PeriodicalId":355632,"journal":{"name":"Proceedings of the 1st International Workshop on Trustworthy AI for Multimedia Computing","volume":"20 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126838537","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

An Empirical Study of Uncertainty Gap for Disentangling Factors 解纠缠因素的不确定性差距实证研究

Proceedings of the 1st International Workshop on Trustworthy AI for Multimedia Computing Pub Date : 2021-10-22 DOI: 10.1145/3475731.3484954

Jiantao Wu, Shentong Mo, Lin Wang

引用次数: 0

Dataset Diversity: Measuring and Mitigating Geographical Bias in Image Search and Retrieval 数据集多样性:测量和减轻图像搜索和检索中的地理偏差

Proceedings of the 1st International Workshop on Trustworthy AI for Multimedia Computing Pub Date : 2021-10-22 DOI: 10.1145/3475731.3484956

Abhishek Mandal, Susan Leavy, S. Little

{"title":"Dataset Diversity: Measuring and Mitigating Geographical Bias in Image Search and Retrieval","authors":"Abhishek Mandal, Susan Leavy, S. Little","doi":"10.1145/3475731.3484956","DOIUrl":"https://doi.org/10.1145/3475731.3484956","url":null,"abstract":"Many popular visual datasets used to train deep neural networksfor computer vision applications, especially for facial analytics,are created by retrieving images from the internet. Search enginesare often used to perform this task. However, due to localisationand personalisation of search results by the search engines alongwith the image indexing method used by these search engines, theresultant images overrepresent the demographics of the region fromwhere they were queried from. As most of the visual datasets arecreated in western countries, they tend to have a western centricbias and when these datasets are used to train deep neural networks,they tend to inherit these biases. Researchers studying the issue ofbias in visual datasets have focused on the racial aspect of thesebiases. We approach this from a geographical perspective. In thispaper, we 1) study how linguistic variations in search queries andgeographical variations in the querying region affect the social andcultural aspects of retrieved images focusing on facial analytics, 2)explore how geographical bias in image search and retrieval cancause racial, cultural and stereotypical bias in visual datasets and3) propose methods to mitigate such biases.","PeriodicalId":355632,"journal":{"name":"Proceedings of the 1st International Workshop on Trustworthy AI for Multimedia Computing","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129847818","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 11

Patch Replacement: A Transformation-based Method to Improve Robustness against Adversarial Attacks 补丁替换:一种基于转换的方法来提高对对抗性攻击的鲁棒性

Proceedings of the 1st International Workshop on Trustworthy AI for Multimedia Computing Pub Date : 2021-10-22 DOI: 10.1145/3475731.3484955

Hanwei Zhang, Yannis Avrithis, T. Furon, L. Amsaleg

{"title":"Patch Replacement: A Transformation-based Method to Improve Robustness against Adversarial Attacks","authors":"Hanwei Zhang, Yannis Avrithis, T. Furon, L. Amsaleg","doi":"10.1145/3475731.3484955","DOIUrl":"https://doi.org/10.1145/3475731.3484955","url":null,"abstract":"Deep Neural Networks (DNNs) are robust against intra-class variability of images, pose variations and random noise, but vulnerable to imperceptible adversarial perturbations that are well-crafted precisely to mislead. While random noise even of relatively large magnitude can hardly affect predictions, adversarial perturbations of very small magnitude can make a classifier fail completely. To enhance robustness, we introduce a new adversarial defense called patch replacement, which transforms both the input images and their intermediate features at early layers to make adversarial perturbations behave similarly to random noise. We decompose images/features into small patches and quantize them according to a codebook learned from legitimate training images. This maintains the semantic information of legitimate images, while removing as much as possible the effect of adversarial perturbations. Experiments show that patch replacement improves robustness against both white-box and gray-box attacks, compared with other transformation-based defenses. It has a low computational cost since it does not need training or fine-tuning the network. Importantly, in the white-box scenario, it increases the robustness, while other transformation-based defenses do not.","PeriodicalId":355632,"journal":{"name":"Proceedings of the 1st International Workshop on Trustworthy AI for Multimedia Computing","volume":"50 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127425149","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1