{"title":"Multimodal GeoAI: An integrated spatio-temporal topic-sentiment model for the analysis of geo-social media posts for disaster management","authors":"David Hanny , Bernd Resch","doi":"10.1016/j.jag.2025.104540","DOIUrl":null,"url":null,"abstract":"<div><div>The analysis of online communication on social networks has become a central research interest to improve disaster management. Especially geo-referenced textual posts have been investigated extensively using techniques from Geospatial Artificial Intelligence (GeoAI), topic modelling and sentiment analysis. However, workflows are traditionally sequential with independent processing steps, limiting their ability to capture interconnections between modalities and risking chains of dependencies. To overcome these limitations, we introduce an integrated GeoAI model called the Joint Spatio-Temporal Topic-Sentiment (JSTTS) model. Our proposed method combines semantic, sentiment, spatial and temporal knowledge into continuous feature vectors and is capable of computing geographically delineated sentiment-associated clusters of topics with meaningful location and temporal information. The properties of the JSTTS model were validated experimentally and the approach was evaluated against a comparable sequential workflow. Overall, our JSTTS model achieved higher topic quality scores with an average of 0.145 compared to 0.034 for the sequential workflow and higher average sentiment uniformity with an average of 0.89 versus 0.73. At the same time, both approaches exhibited similar spatial and temporal variance. As a secondary result, the Geographic Growing Self-Organising Map (Geo-GSOM) was developed to cluster multimodal feature vectors meaningfully in geographic space. It was evaluated on artificial training data where it reproduced up to 90% of the spatial autocorrelation while the non-spatial Growing Self-Organising Map (GSOM) only achieved 55%. The learned neuron grid can also be interpreted geographically. The utility of the JSTTS approach is demonstrated through a case study on the 2021 Ahr Valley flooding in Western Germany, where it identified interpretable multimodal clusters, a subset of which proved relevant for disaster management. The approach can be extended to other use cases and adapted for different modalities, holding potential for numerous follow-up studies.</div></div>","PeriodicalId":73423,"journal":{"name":"International journal of applied earth observation and geoinformation : ITC journal","volume":"139 ","pages":"Article 104540"},"PeriodicalIF":7.6000,"publicationDate":"2025-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"International journal of applied earth observation and geoinformation : ITC journal","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1569843225001876","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"REMOTE SENSING","Score":null,"Total":0}
引用次数: 0
Abstract
The analysis of online communication on social networks has become a central research interest to improve disaster management. Especially geo-referenced textual posts have been investigated extensively using techniques from Geospatial Artificial Intelligence (GeoAI), topic modelling and sentiment analysis. However, workflows are traditionally sequential with independent processing steps, limiting their ability to capture interconnections between modalities and risking chains of dependencies. To overcome these limitations, we introduce an integrated GeoAI model called the Joint Spatio-Temporal Topic-Sentiment (JSTTS) model. Our proposed method combines semantic, sentiment, spatial and temporal knowledge into continuous feature vectors and is capable of computing geographically delineated sentiment-associated clusters of topics with meaningful location and temporal information. The properties of the JSTTS model were validated experimentally and the approach was evaluated against a comparable sequential workflow. Overall, our JSTTS model achieved higher topic quality scores with an average of 0.145 compared to 0.034 for the sequential workflow and higher average sentiment uniformity with an average of 0.89 versus 0.73. At the same time, both approaches exhibited similar spatial and temporal variance. As a secondary result, the Geographic Growing Self-Organising Map (Geo-GSOM) was developed to cluster multimodal feature vectors meaningfully in geographic space. It was evaluated on artificial training data where it reproduced up to 90% of the spatial autocorrelation while the non-spatial Growing Self-Organising Map (GSOM) only achieved 55%. The learned neuron grid can also be interpreted geographically. The utility of the JSTTS approach is demonstrated through a case study on the 2021 Ahr Valley flooding in Western Germany, where it identified interpretable multimodal clusters, a subset of which proved relevant for disaster management. The approach can be extended to other use cases and adapted for different modalities, holding potential for numerous follow-up studies.
期刊介绍:
The International Journal of Applied Earth Observation and Geoinformation publishes original papers that utilize earth observation data for natural resource and environmental inventory and management. These data primarily originate from remote sensing platforms, including satellites and aircraft, supplemented by surface and subsurface measurements. Addressing natural resources such as forests, agricultural land, soils, and water, as well as environmental concerns like biodiversity, land degradation, and hazards, the journal explores conceptual and data-driven approaches. It covers geoinformation themes like capturing, databasing, visualization, interpretation, data quality, and spatial uncertainty.