Shuowei Jin, Francis Y. Yan, Cheng Tan, Anuj Kalia, Xenofon Foukas, Z. Morley Mao
{"title":"AutoSpec:自动生成神经网络规范","authors":"Shuowei Jin, Francis Y. Yan, Cheng Tan, Anuj Kalia, Xenofon Foukas, Z. Morley Mao","doi":"arxiv-2409.10897","DOIUrl":null,"url":null,"abstract":"The increasing adoption of neural networks in learning-augmented systems\nhighlights the importance of model safety and robustness, particularly in\nsafety-critical domains. Despite progress in the formal verification of neural\nnetworks, current practices require users to manually define model\nspecifications -- properties that dictate expected model behavior in various\nscenarios. This manual process, however, is prone to human error, limited in\nscope, and time-consuming. In this paper, we introduce AutoSpec, the first\nframework to automatically generate comprehensive and accurate specifications\nfor neural networks in learning-augmented systems. We also propose the first\nset of metrics for assessing the accuracy and coverage of model specifications,\nestablishing a benchmark for future comparisons. Our evaluation across four\ndistinct applications shows that AutoSpec outperforms human-defined\nspecifications as well as two baseline approaches introduced in this study.","PeriodicalId":501278,"journal":{"name":"arXiv - CS - Software Engineering","volume":"35 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"AutoSpec: Automated Generation of Neural Network Specifications\",\"authors\":\"Shuowei Jin, Francis Y. Yan, Cheng Tan, Anuj Kalia, Xenofon Foukas, Z. Morley Mao\",\"doi\":\"arxiv-2409.10897\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"The increasing adoption of neural networks in learning-augmented systems\\nhighlights the importance of model safety and robustness, particularly in\\nsafety-critical domains. Despite progress in the formal verification of neural\\nnetworks, current practices require users to manually define model\\nspecifications -- properties that dictate expected model behavior in various\\nscenarios. This manual process, however, is prone to human error, limited in\\nscope, and time-consuming. In this paper, we introduce AutoSpec, the first\\nframework to automatically generate comprehensive and accurate specifications\\nfor neural networks in learning-augmented systems. We also propose the first\\nset of metrics for assessing the accuracy and coverage of model specifications,\\nestablishing a benchmark for future comparisons. Our evaluation across four\\ndistinct applications shows that AutoSpec outperforms human-defined\\nspecifications as well as two baseline approaches introduced in this study.\",\"PeriodicalId\":501278,\"journal\":{\"name\":\"arXiv - CS - Software Engineering\",\"volume\":\"35 1\",\"pages\":\"\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-09-17\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - CS - Software Engineering\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2409.10897\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Software Engineering","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.10897","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
AutoSpec: Automated Generation of Neural Network Specifications
The increasing adoption of neural networks in learning-augmented systems
highlights the importance of model safety and robustness, particularly in
safety-critical domains. Despite progress in the formal verification of neural
networks, current practices require users to manually define model
specifications -- properties that dictate expected model behavior in various
scenarios. This manual process, however, is prone to human error, limited in
scope, and time-consuming. In this paper, we introduce AutoSpec, the first
framework to automatically generate comprehensive and accurate specifications
for neural networks in learning-augmented systems. We also propose the first
set of metrics for assessing the accuracy and coverage of model specifications,
establishing a benchmark for future comparisons. Our evaluation across four
distinct applications shows that AutoSpec outperforms human-defined
specifications as well as two baseline approaches introduced in this study.