{"title":"LLM +机器学习优于专家评级,从自我陈述文本预测生活满意度","authors":"Feng Huang;Xia Sun;Aizhu Mei;Yilin Wang;Huimin Ding;Tingshao Zhu","doi":"10.1109/TCSS.2024.3475413","DOIUrl":null,"url":null,"abstract":"This study explores an innovative approach to predicting individual life satisfaction by combining large language models (LLMs) with machine learning (ML) techniques. Traditional life satisfaction assessments rely on self-report questionnaires, which can be time-consuming and resource intensive. To address these limitations, we developed a method that utilizes LLMs for feature extraction from open-ended self-statement texts, followed by ML prediction. We compared this approach with standalone LLM predictions and expert ratings. A sample of 378 participants completed the satisfaction with life scale (SWLS) and wrote self-statements about their current life situation. The LLM-based ML model, using a LightGBM regressor, achieved a correlation of 0.542 with self-reported SWLS scores, outperforming both the standalone LLM (<italic>r</i> <inline-formula><tex-math>$=$</tex-math></inline-formula> 0.491) and expert ratings (<italic>r</i> <inline-formula><tex-math>$=$</tex-math></inline-formula> 0.455). Effect size analysis revealed a statistically significant moderate effect size difference between the LLM-based ML model and expert ratings (Cohen's <italic>d</i> <inline-formula><tex-math>$=$</tex-math></inline-formula> 0.499, 95% CI [0.043, 0.955]). These findings demonstrate the potential of integrating LLM and ML for an efficient and accurate assessment of life satisfaction, challenging conventional methods, and opening new avenues for psychological measurement. The study's implications extend to research, clinical practice, and policymaking, offering promising advancements in AI-assisted psychological assessment.","PeriodicalId":13044,"journal":{"name":"IEEE Transactions on Computational Social Systems","volume":"12 3","pages":"1092-1099"},"PeriodicalIF":4.5000,"publicationDate":"2024-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"LLM Plus Machine Learning Outperform Expert Rating to Predict Life Satisfaction From Self-Statement Text\",\"authors\":\"Feng Huang;Xia Sun;Aizhu Mei;Yilin Wang;Huimin Ding;Tingshao Zhu\",\"doi\":\"10.1109/TCSS.2024.3475413\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"This study explores an innovative approach to predicting individual life satisfaction by combining large language models (LLMs) with machine learning (ML) techniques. Traditional life satisfaction assessments rely on self-report questionnaires, which can be time-consuming and resource intensive. To address these limitations, we developed a method that utilizes LLMs for feature extraction from open-ended self-statement texts, followed by ML prediction. We compared this approach with standalone LLM predictions and expert ratings. A sample of 378 participants completed the satisfaction with life scale (SWLS) and wrote self-statements about their current life situation. The LLM-based ML model, using a LightGBM regressor, achieved a correlation of 0.542 with self-reported SWLS scores, outperforming both the standalone LLM (<italic>r</i> <inline-formula><tex-math>$=$</tex-math></inline-formula> 0.491) and expert ratings (<italic>r</i> <inline-formula><tex-math>$=$</tex-math></inline-formula> 0.455). Effect size analysis revealed a statistically significant moderate effect size difference between the LLM-based ML model and expert ratings (Cohen's <italic>d</i> <inline-formula><tex-math>$=$</tex-math></inline-formula> 0.499, 95% CI [0.043, 0.955]). These findings demonstrate the potential of integrating LLM and ML for an efficient and accurate assessment of life satisfaction, challenging conventional methods, and opening new avenues for psychological measurement. The study's implications extend to research, clinical practice, and policymaking, offering promising advancements in AI-assisted psychological assessment.\",\"PeriodicalId\":13044,\"journal\":{\"name\":\"IEEE Transactions on Computational Social Systems\",\"volume\":\"12 3\",\"pages\":\"1092-1099\"},\"PeriodicalIF\":4.5000,\"publicationDate\":\"2024-10-22\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"IEEE Transactions on Computational Social Systems\",\"FirstCategoryId\":\"94\",\"ListUrlMain\":\"https://ieeexplore.ieee.org/document/10729240/\",\"RegionNum\":2,\"RegionCategory\":\"计算机科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"COMPUTER SCIENCE, CYBERNETICS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Transactions on Computational Social Systems","FirstCategoryId":"94","ListUrlMain":"https://ieeexplore.ieee.org/document/10729240/","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, CYBERNETICS","Score":null,"Total":0}
LLM Plus Machine Learning Outperform Expert Rating to Predict Life Satisfaction From Self-Statement Text
This study explores an innovative approach to predicting individual life satisfaction by combining large language models (LLMs) with machine learning (ML) techniques. Traditional life satisfaction assessments rely on self-report questionnaires, which can be time-consuming and resource intensive. To address these limitations, we developed a method that utilizes LLMs for feature extraction from open-ended self-statement texts, followed by ML prediction. We compared this approach with standalone LLM predictions and expert ratings. A sample of 378 participants completed the satisfaction with life scale (SWLS) and wrote self-statements about their current life situation. The LLM-based ML model, using a LightGBM regressor, achieved a correlation of 0.542 with self-reported SWLS scores, outperforming both the standalone LLM (r $=$ 0.491) and expert ratings (r $=$ 0.455). Effect size analysis revealed a statistically significant moderate effect size difference between the LLM-based ML model and expert ratings (Cohen's d $=$ 0.499, 95% CI [0.043, 0.955]). These findings demonstrate the potential of integrating LLM and ML for an efficient and accurate assessment of life satisfaction, challenging conventional methods, and opening new avenues for psychological measurement. The study's implications extend to research, clinical practice, and policymaking, offering promising advancements in AI-assisted psychological assessment.
期刊介绍:
IEEE Transactions on Computational Social Systems focuses on such topics as modeling, simulation, analysis and understanding of social systems from the quantitative and/or computational perspective. "Systems" include man-man, man-machine and machine-machine organizations and adversarial situations as well as social media structures and their dynamics. More specifically, the proposed transactions publishes articles on modeling the dynamics of social systems, methodologies for incorporating and representing socio-cultural and behavioral aspects in computational modeling, analysis of social system behavior and structure, and paradigms for social systems modeling and simulation. The journal also features articles on social network dynamics, social intelligence and cognition, social systems design and architectures, socio-cultural modeling and representation, and computational behavior modeling, and their applications.