用于开放世界感知的具有互补路径的视觉芯片。

IF 50.5 1区 综合性期刊 Q1 MULTIDISCIPLINARY SCIENCES
Nature Pub Date : 2024-05-29 DOI:10.1038/s41586-024-07358-4
Zheyu Yang, Taoyi Wang, Yihan Lin, Yuguo Chen, Hui Zeng, Jing Pei, Jiazheng Wang, Xue Liu, Yichun Zhou, Jianqiang Zhang, Xin Wang, Xinhao Lv, Rong Zhao, Luping Shi
{"title":"用于开放世界感知的具有互补路径的视觉芯片。","authors":"Zheyu Yang, Taoyi Wang, Yihan Lin, Yuguo Chen, Hui Zeng, Jing Pei, Jiazheng Wang, Xue Liu, Yichun Zhou, Jianqiang Zhang, Xin Wang, Xinhao Lv, Rong Zhao, Luping Shi","doi":"10.1038/s41586-024-07358-4","DOIUrl":null,"url":null,"abstract":"Image sensors face substantial challenges when dealing with dynamic, diverse and unpredictable scenes in open-world applications. However, the development of image sensors towards high speed, high resolution, large dynamic range and high precision is limited by power and bandwidth. Here we present a complementary sensing paradigm inspired by the human visual system that involves parsing visual information into primitive-based representations and assembling these primitives to form two complementary vision pathways: a cognition-oriented pathway for accurate cognition and an action-oriented pathway for rapid response. To realize this paradigm, a vision chip called Tianmouc is developed, incorporating a hybrid pixel array and a parallel-and-heterogeneous readout architecture. Leveraging the characteristics of the complementary vision pathway, Tianmouc achieves high-speed sensing of up to 10,000 fps, a dynamic range of 130 dB and an advanced figure of merit in terms of spatial resolution, speed and dynamic range. Furthermore, it adaptively reduces bandwidth by 90%. We demonstrate the integration of a Tianmouc chip into an autonomous driving system, showcasing its abilities to enable accurate, fast and robust perception, even in challenging corner cases on open roads. The primitive-based complementary sensing paradigm helps in overcoming fundamental limitations in developing vision systems for diverse open-world applications. Inspired by the human visual system, a vision chip with primitive-based complementary pathways is developed to overcome the power and bandwidth wall of vision systems, achieving fast, precise, robust and high-dynamic-range sensing efficiently in the open world.","PeriodicalId":18787,"journal":{"name":"Nature","volume":"629 8014","pages":"1027-1033"},"PeriodicalIF":50.5000,"publicationDate":"2024-05-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"A vision chip with complementary pathways for open-world sensing\",\"authors\":\"Zheyu Yang, Taoyi Wang, Yihan Lin, Yuguo Chen, Hui Zeng, Jing Pei, Jiazheng Wang, Xue Liu, Yichun Zhou, Jianqiang Zhang, Xin Wang, Xinhao Lv, Rong Zhao, Luping Shi\",\"doi\":\"10.1038/s41586-024-07358-4\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Image sensors face substantial challenges when dealing with dynamic, diverse and unpredictable scenes in open-world applications. However, the development of image sensors towards high speed, high resolution, large dynamic range and high precision is limited by power and bandwidth. Here we present a complementary sensing paradigm inspired by the human visual system that involves parsing visual information into primitive-based representations and assembling these primitives to form two complementary vision pathways: a cognition-oriented pathway for accurate cognition and an action-oriented pathway for rapid response. To realize this paradigm, a vision chip called Tianmouc is developed, incorporating a hybrid pixel array and a parallel-and-heterogeneous readout architecture. Leveraging the characteristics of the complementary vision pathway, Tianmouc achieves high-speed sensing of up to 10,000 fps, a dynamic range of 130 dB and an advanced figure of merit in terms of spatial resolution, speed and dynamic range. Furthermore, it adaptively reduces bandwidth by 90%. We demonstrate the integration of a Tianmouc chip into an autonomous driving system, showcasing its abilities to enable accurate, fast and robust perception, even in challenging corner cases on open roads. The primitive-based complementary sensing paradigm helps in overcoming fundamental limitations in developing vision systems for diverse open-world applications. Inspired by the human visual system, a vision chip with primitive-based complementary pathways is developed to overcome the power and bandwidth wall of vision systems, achieving fast, precise, robust and high-dynamic-range sensing efficiently in the open world.\",\"PeriodicalId\":18787,\"journal\":{\"name\":\"Nature\",\"volume\":\"629 8014\",\"pages\":\"1027-1033\"},\"PeriodicalIF\":50.5000,\"publicationDate\":\"2024-05-29\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Nature\",\"FirstCategoryId\":\"103\",\"ListUrlMain\":\"https://www.nature.com/articles/s41586-024-07358-4\",\"RegionNum\":1,\"RegionCategory\":\"综合性期刊\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"MULTIDISCIPLINARY SCIENCES\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Nature","FirstCategoryId":"103","ListUrlMain":"https://www.nature.com/articles/s41586-024-07358-4","RegionNum":1,"RegionCategory":"综合性期刊","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"MULTIDISCIPLINARY SCIENCES","Score":null,"Total":0}
引用次数: 0

摘要

在开放世界应用中,图像传感器在处理动态、多样和不可预测的场景时面临巨大挑战。然而,图像传感器向高速、高分辨率、大动态范围和高精度方向的发展受到了功率和带宽的限制。在这里,我们提出了一种互补传感范式,其灵感来自人类视觉系统,包括将视觉信息解析为基于基元的表征,并将这些基元组合起来,形成两条互补的视觉通路:一条是面向认知的通路,用于准确认知;另一条是面向行动的通路,用于快速反应。为了实现这一范式,我们开发了一种名为 "天目 "的视觉芯片,其中集成了混合像素阵列和并行异构读出架构。利用互补视觉路径的特性,Tianmouc 实现了高达 10,000 fps 的高速传感,130 dB 的动态范围,以及在空间分辨率、速度和动态范围方面的先进优势。此外,它还能自适应地减少 90% 的带宽。我们演示了将天目芯片集成到自动驾驶系统中的过程,展示了其实现准确、快速和稳健感知的能力,即使是在开放道路上具有挑战性的转弯情况下也不例外。基于基元的互补感知范式有助于克服为各种开放世界应用开发视觉系统的基本限制。
本文章由计算机程序翻译,如有差异,请以英文原文为准。

A vision chip with complementary pathways for open-world sensing

A vision chip with complementary pathways for open-world sensing

A vision chip with complementary pathways for open-world sensing
Image sensors face substantial challenges when dealing with dynamic, diverse and unpredictable scenes in open-world applications. However, the development of image sensors towards high speed, high resolution, large dynamic range and high precision is limited by power and bandwidth. Here we present a complementary sensing paradigm inspired by the human visual system that involves parsing visual information into primitive-based representations and assembling these primitives to form two complementary vision pathways: a cognition-oriented pathway for accurate cognition and an action-oriented pathway for rapid response. To realize this paradigm, a vision chip called Tianmouc is developed, incorporating a hybrid pixel array and a parallel-and-heterogeneous readout architecture. Leveraging the characteristics of the complementary vision pathway, Tianmouc achieves high-speed sensing of up to 10,000 fps, a dynamic range of 130 dB and an advanced figure of merit in terms of spatial resolution, speed and dynamic range. Furthermore, it adaptively reduces bandwidth by 90%. We demonstrate the integration of a Tianmouc chip into an autonomous driving system, showcasing its abilities to enable accurate, fast and robust perception, even in challenging corner cases on open roads. The primitive-based complementary sensing paradigm helps in overcoming fundamental limitations in developing vision systems for diverse open-world applications. Inspired by the human visual system, a vision chip with primitive-based complementary pathways is developed to overcome the power and bandwidth wall of vision systems, achieving fast, precise, robust and high-dynamic-range sensing efficiently in the open world.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Nature
Nature 综合性期刊-综合性期刊
CiteScore
90.00
自引率
1.20%
发文量
3652
审稿时长
3 months
期刊介绍: Nature is a prestigious international journal that publishes peer-reviewed research in various scientific and technological fields. The selection of articles is based on criteria such as originality, importance, interdisciplinary relevance, timeliness, accessibility, elegance, and surprising conclusions. In addition to showcasing significant scientific advances, Nature delivers rapid, authoritative, insightful news, and interpretation of current and upcoming trends impacting science, scientists, and the broader public. The journal serves a dual purpose: firstly, to promptly share noteworthy scientific advances and foster discussions among scientists, and secondly, to ensure the swift dissemination of scientific results globally, emphasizing their significance for knowledge, culture, and daily life.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信