{"title":"行动质量评估的层次神经符号法","authors":"Lauren Okamoto, Paritosh Parmar","doi":"arxiv-2403.13798","DOIUrl":null,"url":null,"abstract":"Action quality assessment (AQA) applies computer vision to quantitatively\nassess the performance or execution of a human action. Current AQA approaches\nare end-to-end neural models, which lack transparency and tend to be biased\nbecause they are trained on subjective human judgements as ground-truth. To\naddress these issues, we introduce a neuro-symbolic paradigm for AQA, which\nuses neural networks to abstract interpretable symbols from video data and\nmakes quality assessments by applying rules to those symbols. We take diving as\nthe case study. We found that domain experts prefer our system and find it more\ninformative than purely neural approaches to AQA in diving. Our system also\nachieves state-of-the-art action recognition and temporal segmentation, and\nautomatically generates a detailed report that breaks the dive down into its\nelements and provides objective scoring with visual evidence. As verified by a\ngroup of domain experts, this report may be used to assist judges in scoring,\nhelp train judges, and provide feedback to divers. We will open-source all of\nour annotated training data and code for ease of reproducibility.","PeriodicalId":501033,"journal":{"name":"arXiv - CS - Symbolic Computation","volume":"141 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Hierarchical NeuroSymbolic Approach for Action Quality Assessment\",\"authors\":\"Lauren Okamoto, Paritosh Parmar\",\"doi\":\"arxiv-2403.13798\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Action quality assessment (AQA) applies computer vision to quantitatively\\nassess the performance or execution of a human action. Current AQA approaches\\nare end-to-end neural models, which lack transparency and tend to be biased\\nbecause they are trained on subjective human judgements as ground-truth. To\\naddress these issues, we introduce a neuro-symbolic paradigm for AQA, which\\nuses neural networks to abstract interpretable symbols from video data and\\nmakes quality assessments by applying rules to those symbols. We take diving as\\nthe case study. We found that domain experts prefer our system and find it more\\ninformative than purely neural approaches to AQA in diving. Our system also\\nachieves state-of-the-art action recognition and temporal segmentation, and\\nautomatically generates a detailed report that breaks the dive down into its\\nelements and provides objective scoring with visual evidence. As verified by a\\ngroup of domain experts, this report may be used to assist judges in scoring,\\nhelp train judges, and provide feedback to divers. We will open-source all of\\nour annotated training data and code for ease of reproducibility.\",\"PeriodicalId\":501033,\"journal\":{\"name\":\"arXiv - CS - Symbolic Computation\",\"volume\":\"141 1\",\"pages\":\"\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-03-20\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - CS - Symbolic Computation\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2403.13798\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Symbolic Computation","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2403.13798","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
Hierarchical NeuroSymbolic Approach for Action Quality Assessment
Action quality assessment (AQA) applies computer vision to quantitatively
assess the performance or execution of a human action. Current AQA approaches
are end-to-end neural models, which lack transparency and tend to be biased
because they are trained on subjective human judgements as ground-truth. To
address these issues, we introduce a neuro-symbolic paradigm for AQA, which
uses neural networks to abstract interpretable symbols from video data and
makes quality assessments by applying rules to those symbols. We take diving as
the case study. We found that domain experts prefer our system and find it more
informative than purely neural approaches to AQA in diving. Our system also
achieves state-of-the-art action recognition and temporal segmentation, and
automatically generates a detailed report that breaks the dive down into its
elements and provides objective scoring with visual evidence. As verified by a
group of domain experts, this report may be used to assist judges in scoring,
help train judges, and provide feedback to divers. We will open-source all of
our annotated training data and code for ease of reproducibility.