潜在嵌入:大脑与环境相互作用的基本表征

Brain-X Pub Date : 2023-10-17 DOI:10.1002/brx2.40
Yaning Han, Xiaoting Hou, Chuanliang Han
{"title":"潜在嵌入:大脑与环境相互作用的基本表征","authors":"Yaning Han,&nbsp;Xiaoting Hou,&nbsp;Chuanliang Han","doi":"10.1002/brx2.40","DOIUrl":null,"url":null,"abstract":"<p>The brain governs the behaviors of natural species (including humans and animals), which serves as a central hub integrating incoming sensory signals from the constantly changing environment. Recent cutting-edge technologies in neuroscience from behavioral<span><sup>1</sup></span> and neural levels<span><sup>2</sup></span> have enabled precise and comprehensive measurements. However, the environment–brain–behavior dataset is difficult to interpret because of its high-dimensional nature. To address this challenge, latent embedding has emerged as a promising technique with the property of dimensionality reduction, which can facilitate the identification of common environment–brain–behavior patterns (Figure 1).</p><p>The main idea of extracting latent embeddings is to eliminate dataset redundancy. It requires an algorithm to transform the raw dataset to a new low-dimensional feature space with little information loss. Classically, principal component analysis has been used to linearly transform raw data to an orthogonal space. However, owing to the existence of non-linear structures in nature, the linear transform cannot avoid high information loss in low dimensions. Thus, several non-linear dimensionality reduction methods (t-distributed stochastic neighbor embedding [t-SNE]<span><sup>4</sup></span> and uniform manifold approximation and projection for dimension reduction [UMAP]<span><sup>5</sup></span>) have been developed. However, their non-linear features can reduce the interpretability. For instance, the hippocampus is responsible for representing spatial information and the direction of travel, but pure data-driven latent embeddings (t-SNE or UMAP) may confuse these two functions. These two functions are executed simultaneously, which requires interpretable hypotheses to separate them. Pure data-driven methods cannot introduce existing assumptions to refine latent embeddings. However, using a recent neural network encoder (CEBRA),<span><sup>3</sup></span> this problem can be fully solved. CEBRA addresses this issue by incorporating both supervised and self-supervised learning approaches. By providing supervision through space or direction labels, CEBRA can identify distinct coding patterns in the neural activities of the hippocampus across different latent dimensions, ensuring dimensional alignment with interpretable prior knowledge.</p><p>The main process of CEBRA uses contrastive learning, which was developed to obtain low-dimensional embeddings that are both interpretable and exhibit high performance across various applications.<span><sup>3</sup></span> The contrastive learning technique aims to discover common and distinguishable attributes by contrasting samples, and it optimizes joint latent embeddings from multiple sources, including sensory inputs, brain activities, and behaviors. CEBRA's non-linear encoder combines input data from multiple modalities and uses auxiliary labels to enhance the interpretability. As a result, CEBRA can be applied to both static and dynamic variables, making it a versatile tool for analyzing environment–brain–behavior data. These characteristics enable CEBRA to identify valuable differences across multiple subjects and generate consistent latent embeddings that accurately represent the intrinsic and generalizable information flow across various types of data. This alignment enables CEBRA to accurately predict the locomotion of animals, identify active or passive behavior in primates, and represent stable neural patterns across different recording technologies, subjects, and species using its latent embeddings.</p><p>One amazing result of CEBRA is the reconstruction of videos from mouse visual cortical areas.<span><sup>3</sup></span> Neural activities during natural videos could be encoded in latent embeddings and then decoded with great accuracy. The video has millions of pixel dimensions with temporal dynamics, which have been compressed into neural representations. In this case, CEBRA can further compress them into only three latent dimensions, containing sufficient information to restore the raw videos. These findings demonstrate CEBRA's ability to identify common latent embeddings from visual input to brain activities. Furthermore, they suggest that the brain can compress and process external information in extremely low dimensions. Latent embeddings contain the intricate interactions between the world and the brain without any loss of information.</p><p>Although CEBRA demonstrates the state-of-the-art (SOTA) decoding accuracy of natural videos, the decoding occurs not in the frame contents but in the indexes. Recovering the visual inputs from neural activities is still an issue. One problem is the limitation of the number of recording neurons in current recording technologies, which lose considerable amounts of information. The direction but also the challenge is recording more neuronal activities and their physiological connections. This could reduce the error in estimating neural activity correlations. Another issue is the unclear analytic expression of latent embeddings. This dimness is primarily due to insufficient labeled variables, such as changing edges and instance deformations in videos. In the future, the advancement of neuroethological measurement technologies is essential to further enhance the performance of latent embeddings. Powerful neural networks, such as transformers, may be employed to process increasingly large datasets. An analytical expression is crucial to understand the intricacies of latent embeddings. CEBRA attains interpretability by utilizing low-order variables, such as position and velocity. However, to achieve high-order interpretability, more complex symbolic regression mechanisms are necessary.</p><p>Latent embeddings serve as a crucial intermediary for the transmission of information from the external world to the brain. However, the applications of latent embeddings are not only restricted to the brain. Under the background of clinical big data, latent embeddings have considerable potential for revealing the inner mechanism of disease related to the brain, genes, and other physiological indicators. The complex interactions between drugs and individuals could also be simplified in latent embeddings. Nevertheless, much research is needed to fully understand the underlying meaning of latent embeddings.</p><p><b>Yaning Han</b>, <b>Xiaoting Hou</b>, and <b>Chuanliang Han</b>: Conceptualization; writing – original draft; writing – review &amp; editing.</p><p>The authors declare no competing interests.</p>","PeriodicalId":94303,"journal":{"name":"Brain-X","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2023-10-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/brx2.40","citationCount":"0","resultStr":"{\"title\":\"Latent embeddings: An essential representation of brain–environment interactions\",\"authors\":\"Yaning Han,&nbsp;Xiaoting Hou,&nbsp;Chuanliang Han\",\"doi\":\"10.1002/brx2.40\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p>The brain governs the behaviors of natural species (including humans and animals), which serves as a central hub integrating incoming sensory signals from the constantly changing environment. Recent cutting-edge technologies in neuroscience from behavioral<span><sup>1</sup></span> and neural levels<span><sup>2</sup></span> have enabled precise and comprehensive measurements. However, the environment–brain–behavior dataset is difficult to interpret because of its high-dimensional nature. To address this challenge, latent embedding has emerged as a promising technique with the property of dimensionality reduction, which can facilitate the identification of common environment–brain–behavior patterns (Figure 1).</p><p>The main idea of extracting latent embeddings is to eliminate dataset redundancy. It requires an algorithm to transform the raw dataset to a new low-dimensional feature space with little information loss. Classically, principal component analysis has been used to linearly transform raw data to an orthogonal space. However, owing to the existence of non-linear structures in nature, the linear transform cannot avoid high information loss in low dimensions. Thus, several non-linear dimensionality reduction methods (t-distributed stochastic neighbor embedding [t-SNE]<span><sup>4</sup></span> and uniform manifold approximation and projection for dimension reduction [UMAP]<span><sup>5</sup></span>) have been developed. However, their non-linear features can reduce the interpretability. For instance, the hippocampus is responsible for representing spatial information and the direction of travel, but pure data-driven latent embeddings (t-SNE or UMAP) may confuse these two functions. These two functions are executed simultaneously, which requires interpretable hypotheses to separate them. Pure data-driven methods cannot introduce existing assumptions to refine latent embeddings. However, using a recent neural network encoder (CEBRA),<span><sup>3</sup></span> this problem can be fully solved. CEBRA addresses this issue by incorporating both supervised and self-supervised learning approaches. By providing supervision through space or direction labels, CEBRA can identify distinct coding patterns in the neural activities of the hippocampus across different latent dimensions, ensuring dimensional alignment with interpretable prior knowledge.</p><p>The main process of CEBRA uses contrastive learning, which was developed to obtain low-dimensional embeddings that are both interpretable and exhibit high performance across various applications.<span><sup>3</sup></span> The contrastive learning technique aims to discover common and distinguishable attributes by contrasting samples, and it optimizes joint latent embeddings from multiple sources, including sensory inputs, brain activities, and behaviors. CEBRA's non-linear encoder combines input data from multiple modalities and uses auxiliary labels to enhance the interpretability. As a result, CEBRA can be applied to both static and dynamic variables, making it a versatile tool for analyzing environment–brain–behavior data. These characteristics enable CEBRA to identify valuable differences across multiple subjects and generate consistent latent embeddings that accurately represent the intrinsic and generalizable information flow across various types of data. This alignment enables CEBRA to accurately predict the locomotion of animals, identify active or passive behavior in primates, and represent stable neural patterns across different recording technologies, subjects, and species using its latent embeddings.</p><p>One amazing result of CEBRA is the reconstruction of videos from mouse visual cortical areas.<span><sup>3</sup></span> Neural activities during natural videos could be encoded in latent embeddings and then decoded with great accuracy. The video has millions of pixel dimensions with temporal dynamics, which have been compressed into neural representations. In this case, CEBRA can further compress them into only three latent dimensions, containing sufficient information to restore the raw videos. These findings demonstrate CEBRA's ability to identify common latent embeddings from visual input to brain activities. Furthermore, they suggest that the brain can compress and process external information in extremely low dimensions. Latent embeddings contain the intricate interactions between the world and the brain without any loss of information.</p><p>Although CEBRA demonstrates the state-of-the-art (SOTA) decoding accuracy of natural videos, the decoding occurs not in the frame contents but in the indexes. Recovering the visual inputs from neural activities is still an issue. One problem is the limitation of the number of recording neurons in current recording technologies, which lose considerable amounts of information. The direction but also the challenge is recording more neuronal activities and their physiological connections. This could reduce the error in estimating neural activity correlations. Another issue is the unclear analytic expression of latent embeddings. This dimness is primarily due to insufficient labeled variables, such as changing edges and instance deformations in videos. In the future, the advancement of neuroethological measurement technologies is essential to further enhance the performance of latent embeddings. Powerful neural networks, such as transformers, may be employed to process increasingly large datasets. An analytical expression is crucial to understand the intricacies of latent embeddings. CEBRA attains interpretability by utilizing low-order variables, such as position and velocity. However, to achieve high-order interpretability, more complex symbolic regression mechanisms are necessary.</p><p>Latent embeddings serve as a crucial intermediary for the transmission of information from the external world to the brain. However, the applications of latent embeddings are not only restricted to the brain. Under the background of clinical big data, latent embeddings have considerable potential for revealing the inner mechanism of disease related to the brain, genes, and other physiological indicators. The complex interactions between drugs and individuals could also be simplified in latent embeddings. Nevertheless, much research is needed to fully understand the underlying meaning of latent embeddings.</p><p><b>Yaning Han</b>, <b>Xiaoting Hou</b>, and <b>Chuanliang Han</b>: Conceptualization; writing – original draft; writing – review &amp; editing.</p><p>The authors declare no competing interests.</p>\",\"PeriodicalId\":94303,\"journal\":{\"name\":\"Brain-X\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2023-10-17\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://onlinelibrary.wiley.com/doi/epdf/10.1002/brx2.40\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Brain-X\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://onlinelibrary.wiley.com/doi/10.1002/brx2.40\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Brain-X","FirstCategoryId":"1085","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1002/brx2.40","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

摘要

大脑控制着自然物种(包括人类和动物)的行为,它是整合不断变化的环境中传入的感官信号的中心枢纽。从行为1和神经层面2的神经科学最新前沿技术已经实现了精确和全面的测量。然而,环境-大脑-行为数据集由于其高维性质而难以解释。为了应对这一挑战,潜在嵌入已成为一种具有降维特性的有前途的技术,它可以促进常见环境-大脑-行为模式的识别(图1)。提取潜在嵌入的主要思想是消除数据集冗余。它需要一种算法来将原始数据集转换到一个新的低维特征空间,而信息损失很小。传统上,主成分分析用于将原始数据线性变换到正交空间。然而,由于自然界中存在非线性结构,线性变换无法避免低维的高信息损失。因此,已经开发了几种非线性降维方法(t-分布随机邻域嵌入[t-SNE]4和降维的一致流形逼近和投影[UMAP]5)。然而,它们的非线性特征会降低可解释性。例如,海马体负责表示空间信息和行进方向,但纯数据驱动的潜在嵌入(t-SNE或UMAP)可能会混淆这两种功能。这两个函数同时执行,这需要可解释的假设来将它们分开。纯数据驱动的方法不能引入现有的假设来细化潜在的嵌入。然而,使用最近的神经网络编码器(CEBRA),3这个问题可以完全解决。CEBRA通过结合监督和自我监督学习方法来解决这个问题。通过空间或方向标签提供监督,CEBRA可以在不同的潜在维度上识别海马神经活动的不同编码模式,确保维度与可解释的先验知识一致。CEBRA的主要过程使用对比学习,该学习旨在获得可解释且在各种应用中表现出高性能的低维嵌入。3对比学习技术旨在通过对比样本发现共同和可区分的属性,并优化来自多个来源的联合潜在嵌入,包括感觉输入,大脑活动和行为。CEBRA的非线性编码器结合了来自多种模态的输入数据,并使用辅助标签来增强可解释性。因此,CEBRA可以应用于静态和动态变量,使其成为分析环境-大脑-行为数据的通用工具。这些特征使CEBRA能够识别多个主题之间的有价值差异,并生成一致的潜在嵌入,准确地表示各种类型数据之间的内在和可推广的信息流。这种对齐使CEBRA能够准确预测动物的运动,识别灵长类动物的主动或被动行为,并使用其潜在嵌入来代表不同记录技术、受试者和物种的稳定神经模式。CEBRA的一个惊人结果是从小鼠视觉皮层区域重建视频。3自然视频中的神经活动可以在潜在嵌入中编码,然后以非常高的精度解码。该视频具有数百万个具有时间动态的像素维度,这些像素维度已被压缩为神经表示。在这种情况下,CEBRA可以将它们进一步压缩为三个潜在的维度,其中包含足够的信息来恢复原始视频。这些发现证明了CEBRA识别从视觉输入到大脑活动的常见潜在嵌入的能力。此外,他们认为大脑可以在极低的维度上压缩和处理外部信息。潜在嵌入包含了世界和大脑之间复杂的互动,而不会丢失任何信息。尽管CEBRA展示了自然视频的最先进(SOTA)解码精度,但解码并不发生在帧内容中,而是发生在索引中。从神经活动中恢复视觉输入仍然是一个问题。一个问题是当前记录技术中记录神经元数量的限制,这会损失大量信息。方向也是挑战是记录更多的神经元活动及其生理联系。这可以减少估计神经活动相关性的误差。另一个问题是潜在嵌入的不明确的分析表达。
本文章由计算机程序翻译,如有差异,请以英文原文为准。

Latent embeddings: An essential representation of brain–environment interactions

Latent embeddings: An essential representation of brain–environment interactions

The brain governs the behaviors of natural species (including humans and animals), which serves as a central hub integrating incoming sensory signals from the constantly changing environment. Recent cutting-edge technologies in neuroscience from behavioral1 and neural levels2 have enabled precise and comprehensive measurements. However, the environment–brain–behavior dataset is difficult to interpret because of its high-dimensional nature. To address this challenge, latent embedding has emerged as a promising technique with the property of dimensionality reduction, which can facilitate the identification of common environment–brain–behavior patterns (Figure 1).

The main idea of extracting latent embeddings is to eliminate dataset redundancy. It requires an algorithm to transform the raw dataset to a new low-dimensional feature space with little information loss. Classically, principal component analysis has been used to linearly transform raw data to an orthogonal space. However, owing to the existence of non-linear structures in nature, the linear transform cannot avoid high information loss in low dimensions. Thus, several non-linear dimensionality reduction methods (t-distributed stochastic neighbor embedding [t-SNE]4 and uniform manifold approximation and projection for dimension reduction [UMAP]5) have been developed. However, their non-linear features can reduce the interpretability. For instance, the hippocampus is responsible for representing spatial information and the direction of travel, but pure data-driven latent embeddings (t-SNE or UMAP) may confuse these two functions. These two functions are executed simultaneously, which requires interpretable hypotheses to separate them. Pure data-driven methods cannot introduce existing assumptions to refine latent embeddings. However, using a recent neural network encoder (CEBRA),3 this problem can be fully solved. CEBRA addresses this issue by incorporating both supervised and self-supervised learning approaches. By providing supervision through space or direction labels, CEBRA can identify distinct coding patterns in the neural activities of the hippocampus across different latent dimensions, ensuring dimensional alignment with interpretable prior knowledge.

The main process of CEBRA uses contrastive learning, which was developed to obtain low-dimensional embeddings that are both interpretable and exhibit high performance across various applications.3 The contrastive learning technique aims to discover common and distinguishable attributes by contrasting samples, and it optimizes joint latent embeddings from multiple sources, including sensory inputs, brain activities, and behaviors. CEBRA's non-linear encoder combines input data from multiple modalities and uses auxiliary labels to enhance the interpretability. As a result, CEBRA can be applied to both static and dynamic variables, making it a versatile tool for analyzing environment–brain–behavior data. These characteristics enable CEBRA to identify valuable differences across multiple subjects and generate consistent latent embeddings that accurately represent the intrinsic and generalizable information flow across various types of data. This alignment enables CEBRA to accurately predict the locomotion of animals, identify active or passive behavior in primates, and represent stable neural patterns across different recording technologies, subjects, and species using its latent embeddings.

One amazing result of CEBRA is the reconstruction of videos from mouse visual cortical areas.3 Neural activities during natural videos could be encoded in latent embeddings and then decoded with great accuracy. The video has millions of pixel dimensions with temporal dynamics, which have been compressed into neural representations. In this case, CEBRA can further compress them into only three latent dimensions, containing sufficient information to restore the raw videos. These findings demonstrate CEBRA's ability to identify common latent embeddings from visual input to brain activities. Furthermore, they suggest that the brain can compress and process external information in extremely low dimensions. Latent embeddings contain the intricate interactions between the world and the brain without any loss of information.

Although CEBRA demonstrates the state-of-the-art (SOTA) decoding accuracy of natural videos, the decoding occurs not in the frame contents but in the indexes. Recovering the visual inputs from neural activities is still an issue. One problem is the limitation of the number of recording neurons in current recording technologies, which lose considerable amounts of information. The direction but also the challenge is recording more neuronal activities and their physiological connections. This could reduce the error in estimating neural activity correlations. Another issue is the unclear analytic expression of latent embeddings. This dimness is primarily due to insufficient labeled variables, such as changing edges and instance deformations in videos. In the future, the advancement of neuroethological measurement technologies is essential to further enhance the performance of latent embeddings. Powerful neural networks, such as transformers, may be employed to process increasingly large datasets. An analytical expression is crucial to understand the intricacies of latent embeddings. CEBRA attains interpretability by utilizing low-order variables, such as position and velocity. However, to achieve high-order interpretability, more complex symbolic regression mechanisms are necessary.

Latent embeddings serve as a crucial intermediary for the transmission of information from the external world to the brain. However, the applications of latent embeddings are not only restricted to the brain. Under the background of clinical big data, latent embeddings have considerable potential for revealing the inner mechanism of disease related to the brain, genes, and other physiological indicators. The complex interactions between drugs and individuals could also be simplified in latent embeddings. Nevertheless, much research is needed to fully understand the underlying meaning of latent embeddings.

Yaning Han, Xiaoting Hou, and Chuanliang Han: Conceptualization; writing – original draft; writing – review & editing.

The authors declare no competing interests.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信