Montgomery Flora, Burkely T. Gallo, Corey K. Potvin, Adam J. Clark, Katie A. Wilson
{"title":"探索预报预警系统中机器学习恶劣天气指导的实用性:2022 年 NOAA 危险天气试验台春季预报实验结果","authors":"Montgomery Flora, Burkely T. Gallo, Corey K. Potvin, Adam J. Clark, Katie A. Wilson","doi":"10.1175/waf-d-24-0038.1","DOIUrl":null,"url":null,"abstract":"\nArtificial intelligence (AI) is gaining popularity for severe weather forecasting. Recently, the authors developed an AI system using machine learning (ML) to produce probabilistic guidance for severe weather hazards, including tornadoes, large hail, and severe winds, using the National Severe Storms Laboratory’s (NSSL) Warn-on-Forecast System as input (WoFS). Known as WoFS-ML-Severe, it performed well in retrospective cases, but its operational usefulness had yet to be determined. To examine the potential usefulness of the ML guidance, we conducted a control and treatment (experimental) group experiment during the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment (HWT-SFE). The control group had full access to WoFS, while the experimental group had access to WoFS and ML products. Explainability graphics were also integrated into the WoFS web viewer. Both groups issued 1-hr convective outlooks for each hazard. After issuing their forecasts, we surveyed participants on their confidence, the number of products viewed, and the usefulness of the ML guidance. We found the ML-based outlooks outperformed non-ML-based outlooks for multiple verification metrics for all three hazards and were rated subjectively higher by the participants. However, the difference in confidence between the two groups was not significant, and the experimental group self-reported viewing more products than the control group. Participants had mixed sentiments towards explainability products as it improved their understanding of the input/output relationships, but viewing them added to their workload. Although the experiment demonstrated the usefulness of ML guidance for severe weather forecasting, there are avenues to improve upon the ML guidance, and more training and exposure are needed to exploit its benefits fully.","PeriodicalId":49369,"journal":{"name":"Weather and Forecasting","volume":null,"pages":null},"PeriodicalIF":3.0000,"publicationDate":"2024-05-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Exploring the Usefulness of Machine Learning Severe Weather Guidance in the Warn-on-Forecast System: Results from the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment\",\"authors\":\"Montgomery Flora, Burkely T. Gallo, Corey K. Potvin, Adam J. Clark, Katie A. Wilson\",\"doi\":\"10.1175/waf-d-24-0038.1\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"\\nArtificial intelligence (AI) is gaining popularity for severe weather forecasting. Recently, the authors developed an AI system using machine learning (ML) to produce probabilistic guidance for severe weather hazards, including tornadoes, large hail, and severe winds, using the National Severe Storms Laboratory’s (NSSL) Warn-on-Forecast System as input (WoFS). Known as WoFS-ML-Severe, it performed well in retrospective cases, but its operational usefulness had yet to be determined. To examine the potential usefulness of the ML guidance, we conducted a control and treatment (experimental) group experiment during the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment (HWT-SFE). The control group had full access to WoFS, while the experimental group had access to WoFS and ML products. Explainability graphics were also integrated into the WoFS web viewer. Both groups issued 1-hr convective outlooks for each hazard. After issuing their forecasts, we surveyed participants on their confidence, the number of products viewed, and the usefulness of the ML guidance. We found the ML-based outlooks outperformed non-ML-based outlooks for multiple verification metrics for all three hazards and were rated subjectively higher by the participants. However, the difference in confidence between the two groups was not significant, and the experimental group self-reported viewing more products than the control group. Participants had mixed sentiments towards explainability products as it improved their understanding of the input/output relationships, but viewing them added to their workload. Although the experiment demonstrated the usefulness of ML guidance for severe weather forecasting, there are avenues to improve upon the ML guidance, and more training and exposure are needed to exploit its benefits fully.\",\"PeriodicalId\":49369,\"journal\":{\"name\":\"Weather and Forecasting\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":3.0000,\"publicationDate\":\"2024-05-13\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Weather and Forecasting\",\"FirstCategoryId\":\"89\",\"ListUrlMain\":\"https://doi.org/10.1175/waf-d-24-0038.1\",\"RegionNum\":3,\"RegionCategory\":\"地球科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"METEOROLOGY & ATMOSPHERIC SCIENCES\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Weather and Forecasting","FirstCategoryId":"89","ListUrlMain":"https://doi.org/10.1175/waf-d-24-0038.1","RegionNum":3,"RegionCategory":"地球科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"METEOROLOGY & ATMOSPHERIC SCIENCES","Score":null,"Total":0}
引用次数: 0
摘要
人工智能(AI)在恶劣天气预报中越来越受欢迎。最近,作者利用机器学习(ML)开发了一套人工智能系统,以国家强风暴实验室(NSSL)的预警预报系统(WoFS)为输入,为龙卷风、大冰雹和强风等恶劣天气灾害提供概率指导。该系统被称为 WoFS-ML-Severe,在回顾性案例中表现良好,但其实用性尚待确定。为了检验 ML 指导的潜在实用性,我们在 2022 年 NOAA 危险天气试验台春季预报实验(HWT-SFE)期间进行了对照组和处理(实验)组实验。对照组可以完全访问 WoFS,而实验组则可以访问 WoFS 和 ML 产品。可解释性图形也被集成到 WoFS 网页浏览器中。两组都对每种灾害发布了 1 小时的对流展望。发布预报后,我们对参与者的信心、查看的产品数量以及 ML 指导的实用性进行了调查。我们发现,在所有三种灾害的多个验证指标上,基于 ML 的展望都优于非基于 ML 的展望,而且参与者的主观评价也更高。然而,两组参与者在信心方面的差异并不显著,而且实验组比对照组自我报告查看了更多的产品。参与者对可解释性产品的评价不一,因为它提高了他们对输入/输出关系的理解,但观看这些产品却增加了他们的工作量。尽管实验证明了可解释性指导对恶劣天气预报的有用性,但仍有改进可解释性指导的途径,而且需要更多的培训和接触才能充分发挥其优势。
Exploring the Usefulness of Machine Learning Severe Weather Guidance in the Warn-on-Forecast System: Results from the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment
Artificial intelligence (AI) is gaining popularity for severe weather forecasting. Recently, the authors developed an AI system using machine learning (ML) to produce probabilistic guidance for severe weather hazards, including tornadoes, large hail, and severe winds, using the National Severe Storms Laboratory’s (NSSL) Warn-on-Forecast System as input (WoFS). Known as WoFS-ML-Severe, it performed well in retrospective cases, but its operational usefulness had yet to be determined. To examine the potential usefulness of the ML guidance, we conducted a control and treatment (experimental) group experiment during the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment (HWT-SFE). The control group had full access to WoFS, while the experimental group had access to WoFS and ML products. Explainability graphics were also integrated into the WoFS web viewer. Both groups issued 1-hr convective outlooks for each hazard. After issuing their forecasts, we surveyed participants on their confidence, the number of products viewed, and the usefulness of the ML guidance. We found the ML-based outlooks outperformed non-ML-based outlooks for multiple verification metrics for all three hazards and were rated subjectively higher by the participants. However, the difference in confidence between the two groups was not significant, and the experimental group self-reported viewing more products than the control group. Participants had mixed sentiments towards explainability products as it improved their understanding of the input/output relationships, but viewing them added to their workload. Although the experiment demonstrated the usefulness of ML guidance for severe weather forecasting, there are avenues to improve upon the ML guidance, and more training and exposure are needed to exploit its benefits fully.
期刊介绍:
Weather and Forecasting (WAF) (ISSN: 0882-8156; eISSN: 1520-0434) publishes research that is relevant to operational forecasting. This includes papers on significant weather events, forecasting techniques, forecast verification, model parameterizations, data assimilation, model ensembles, statistical postprocessing techniques, the transfer of research results to the forecasting community, and the societal use and value of forecasts. The scope of WAF includes research relevant to forecast lead times ranging from short-term “nowcasts” through seasonal time scales out to approximately two years.