Self-Attention Based Semantic Decomposition in Vector Symbolic Architectures

Calvin Yeung, Prathyush Poduval, Mohsen Imani
{"title":"Self-Attention Based Semantic Decomposition in Vector Symbolic Architectures","authors":"Calvin Yeung, Prathyush Poduval, Mohsen Imani","doi":"arxiv-2403.13218","DOIUrl":null,"url":null,"abstract":"Vector Symbolic Architectures (VSAs) have emerged as a novel framework for\nenabling interpretable machine learning algorithms equipped with the ability to\nreason and explain their decision processes. The basic idea is to represent\ndiscrete information through high dimensional random vectors. Complex data\nstructures can be built up with operations over vectors such as the \"binding\"\noperation involving element-wise vector multiplication, which associates data\ntogether. The reverse task of decomposing the associated elements is a\ncombinatorially hard task, with an exponentially large search space. The main\nalgorithm for performing this search is the resonator network, inspired by\nHopfield network-based memory search operations. In this work, we introduce a new variant of the resonator network, based on\nself-attention based update rules in the iterative search problem. This update\nrule, based on the Hopfield network with log-sum-exp energy function and\nnorm-bounded states, is shown to substantially improve the performance and rate\nof convergence. As a result, our algorithm enables a larger capacity for\nassociative memory, enabling applications in many tasks like perception based\npattern recognition, scene decomposition, and object reasoning. We substantiate\nour algorithm with a thorough evaluation and comparisons to baselines.","PeriodicalId":501033,"journal":{"name":"arXiv - CS - Symbolic Computation","volume":"42 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Symbolic Computation","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2403.13218","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

Vector Symbolic Architectures (VSAs) have emerged as a novel framework for enabling interpretable machine learning algorithms equipped with the ability to reason and explain their decision processes. The basic idea is to represent discrete information through high dimensional random vectors. Complex data structures can be built up with operations over vectors such as the "binding" operation involving element-wise vector multiplication, which associates data together. The reverse task of decomposing the associated elements is a combinatorially hard task, with an exponentially large search space. The main algorithm for performing this search is the resonator network, inspired by Hopfield network-based memory search operations. In this work, we introduce a new variant of the resonator network, based on self-attention based update rules in the iterative search problem. This update rule, based on the Hopfield network with log-sum-exp energy function and norm-bounded states, is shown to substantially improve the performance and rate of convergence. As a result, our algorithm enables a larger capacity for associative memory, enabling applications in many tasks like perception based pattern recognition, scene decomposition, and object reasoning. We substantiate our algorithm with a thorough evaluation and comparisons to baselines.
矢量符号架构中基于自注意力的语义分解
矢量符号架构(VSA)是一种新颖的框架,它使可解释的机器学习算法具备了推理和解释其决策过程的能力。其基本思想是通过高维随机向量来表示离散信息。复杂的数据结构可以通过对向量的运算建立起来,例如 "绑定 "运算(涉及元素与向量相乘),它将数据关联在一起。分解关联元素的逆向任务是一项难以组合的任务,其搜索空间呈指数级增长。执行这种搜索的主要算法是共振网络,其灵感来自基于霍普菲尔德网络的内存搜索操作。在这项工作中,我们在迭代搜索问题中引入了一种基于自我关注更新规则的共振网络新变体。这种更新规则基于具有对数求和-展开能量函数和规范约束状态的霍普菲尔德网络,其性能和收敛速度都得到了大幅提高。因此,我们的算法可以实现更大的关联记忆容量,从而可以应用于许多任务中,如基于感知的模式识别、场景分解和对象推理。我们通过全面的评估和与基线的比较证实了我们的算法。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信