Multimodal GeoAI: An integrated spatio-temporal topic-sentiment model for the analysis of geo-social media posts for disaster management

IF 7.6 Q1 REMOTE SENSING
David Hanny , Bernd Resch
{"title":"Multimodal GeoAI: An integrated spatio-temporal topic-sentiment model for the analysis of geo-social media posts for disaster management","authors":"David Hanny ,&nbsp;Bernd Resch","doi":"10.1016/j.jag.2025.104540","DOIUrl":null,"url":null,"abstract":"<div><div>The analysis of online communication on social networks has become a central research interest to improve disaster management. Especially geo-referenced textual posts have been investigated extensively using techniques from Geospatial Artificial Intelligence (GeoAI), topic modelling and sentiment analysis. However, workflows are traditionally sequential with independent processing steps, limiting their ability to capture interconnections between modalities and risking chains of dependencies. To overcome these limitations, we introduce an integrated GeoAI model called the Joint Spatio-Temporal Topic-Sentiment (JSTTS) model. Our proposed method combines semantic, sentiment, spatial and temporal knowledge into continuous feature vectors and is capable of computing geographically delineated sentiment-associated clusters of topics with meaningful location and temporal information. The properties of the JSTTS model were validated experimentally and the approach was evaluated against a comparable sequential workflow. Overall, our JSTTS model achieved higher topic quality scores with an average of 0.145 compared to 0.034 for the sequential workflow and higher average sentiment uniformity with an average of 0.89 versus 0.73. At the same time, both approaches exhibited similar spatial and temporal variance. As a secondary result, the Geographic Growing Self-Organising Map (Geo-GSOM) was developed to cluster multimodal feature vectors meaningfully in geographic space. It was evaluated on artificial training data where it reproduced up to 90% of the spatial autocorrelation while the non-spatial Growing Self-Organising Map (GSOM) only achieved 55%. The learned neuron grid can also be interpreted geographically. The utility of the JSTTS approach is demonstrated through a case study on the 2021 Ahr Valley flooding in Western Germany, where it identified interpretable multimodal clusters, a subset of which proved relevant for disaster management. The approach can be extended to other use cases and adapted for different modalities, holding potential for numerous follow-up studies.</div></div>","PeriodicalId":73423,"journal":{"name":"International journal of applied earth observation and geoinformation : ITC journal","volume":"139 ","pages":"Article 104540"},"PeriodicalIF":7.6000,"publicationDate":"2025-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"International journal of applied earth observation and geoinformation : ITC journal","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1569843225001876","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"REMOTE SENSING","Score":null,"Total":0}
引用次数: 0

Abstract

The analysis of online communication on social networks has become a central research interest to improve disaster management. Especially geo-referenced textual posts have been investigated extensively using techniques from Geospatial Artificial Intelligence (GeoAI), topic modelling and sentiment analysis. However, workflows are traditionally sequential with independent processing steps, limiting their ability to capture interconnections between modalities and risking chains of dependencies. To overcome these limitations, we introduce an integrated GeoAI model called the Joint Spatio-Temporal Topic-Sentiment (JSTTS) model. Our proposed method combines semantic, sentiment, spatial and temporal knowledge into continuous feature vectors and is capable of computing geographically delineated sentiment-associated clusters of topics with meaningful location and temporal information. The properties of the JSTTS model were validated experimentally and the approach was evaluated against a comparable sequential workflow. Overall, our JSTTS model achieved higher topic quality scores with an average of 0.145 compared to 0.034 for the sequential workflow and higher average sentiment uniformity with an average of 0.89 versus 0.73. At the same time, both approaches exhibited similar spatial and temporal variance. As a secondary result, the Geographic Growing Self-Organising Map (Geo-GSOM) was developed to cluster multimodal feature vectors meaningfully in geographic space. It was evaluated on artificial training data where it reproduced up to 90% of the spatial autocorrelation while the non-spatial Growing Self-Organising Map (GSOM) only achieved 55%. The learned neuron grid can also be interpreted geographically. The utility of the JSTTS approach is demonstrated through a case study on the 2021 Ahr Valley flooding in Western Germany, where it identified interpretable multimodal clusters, a subset of which proved relevant for disaster management. The approach can be extended to other use cases and adapted for different modalities, holding potential for numerous follow-up studies.

Abstract Image

Multimodal GeoAI:用于灾害管理的地理社交媒体帖子分析的综合时空主题情绪模型
对社交网络上的在线交流进行分析已经成为改善灾害管理的一个核心研究兴趣。特别是地理空间人工智能(GeoAI)、主题建模和情感分析等技术对地理参考文本帖子进行了广泛的研究。然而,工作流传统上是顺序的,具有独立的处理步骤,限制了它们捕获模式之间的互连和依赖关系的风险链的能力。为了克服这些限制,我们引入了一种称为联合时空主题情感(JSTTS)模型的集成GeoAI模型。我们提出的方法将语义、情感、空间和时间知识结合成连续的特征向量,并能够计算具有有意义的位置和时间信息的地理描述的主题情感相关聚类。通过实验验证了JSTTS模型的特性,并对一个可比较的顺序工作流进行了评估。总体而言,我们的JSTTS模型获得了更高的主题质量得分,平均为0.145,而顺序工作流的平均得分为0.034,平均情绪一致性更高,平均为0.89,而顺序工作流的平均得分为0.73。同时,两种方法均表现出相似的时空变异。其次,开发了地理生长自组织图(Geo-GSOM),用于在地理空间中对多模态特征向量进行有意义的聚类。在人工训练数据上对其进行了评估,其中再现了高达90%的空间自相关,而非空间生长自组织图(GSOM)仅实现了55%。学习到的神经元网格也可以在地理上解释。通过对2021年德国西部Ahr谷洪水的案例研究,JSTTS方法的实用性得到了证明,该方法确定了可解释的多模态集群,其中一个子集被证明与灾害管理相关。该方法可以扩展到其他用例并适用于不同的模式,具有大量后续研究的潜力。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
International journal of applied earth observation and geoinformation : ITC journal
International journal of applied earth observation and geoinformation : ITC journal Global and Planetary Change, Management, Monitoring, Policy and Law, Earth-Surface Processes, Computers in Earth Sciences
CiteScore
12.00
自引率
0.00%
发文量
0
审稿时长
77 days
期刊介绍: The International Journal of Applied Earth Observation and Geoinformation publishes original papers that utilize earth observation data for natural resource and environmental inventory and management. These data primarily originate from remote sensing platforms, including satellites and aircraft, supplemented by surface and subsurface measurements. Addressing natural resources such as forests, agricultural land, soils, and water, as well as environmental concerns like biodiversity, land degradation, and hazards, the journal explores conceptual and data-driven approaches. It covers geoinformation themes like capturing, databasing, visualization, interpretation, data quality, and spatial uncertainty.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信