Validation of a generative artificial intelligence tool for the critical appraisal of articles on the epidemiology of mental health: Its application in the Middle East and North Africa.
{"title":"Validation of a generative artificial intelligence tool for the critical appraisal of articles on the epidemiology of mental health: Its application in the Middle East and North Africa.","authors":"Moussa Cheima, Altayyar Sarah, Vergonjeanne Marion, Gelle Thibaut, Preux Pierre-Marie","doi":"10.1016/j.jeph.2025.202990","DOIUrl":null,"url":null,"abstract":"<p><p>Mental health disorders have a high disability-adjusted life years in the Middle East and North Africa. This rise has led to a surge in related publications, prompting researchers to use AI tools like ChatGPT to reduce time spent on routine tasks. Our study aimed to validate an AI-assisted critical appraisal (CA) tool by comparing it with human raters. We developed customized GPT models using ChatGPT-4. These models were tailored to evaluate studies using the Newcastle-Ottawa Scale (NOS) or the Jadad Scale in one model, while another model evaluated STROBE or CONSORT guidelines. Our results showed a moderate to good agreement between human CA and our GPTs for the NOS for cohort, case control and cross-sectional studies and for the Jadad scale, with an ICC of 0.68 [95 %CI: 0.24-0.82], 0.69 [95 %CI: 0.31-0.88], 0.76 [95 %CI: 0.47-0.90] and 0.84 [95 %CI: 0.57-0.94] respectively. There was also a moderate to substantial agreement between the two methods for STROBE in cross sectional, cohort, case control studies, and for CONSORT in trial design, with a K of 0.63 [95 %CI: 0.56-0.70], 0.57 [95 %CI: 0.47-0.66], 0.48 [95 %CI: 0.38-0.50] and 0.70 [95 %CI: 0.63-0.77] respectively. Our custom GPT models produced hallucinations in 6.5 % and 4.9 % of cases, respectively. Human raters took an average of 19.6 ± 4.3 min per article, whereas our customized GPTs took only 1.4. ChatGPT could be a useful tool for handling repetitive tasks yet its effective application relies on the critical expertise of researchers.</p>","PeriodicalId":517428,"journal":{"name":"Journal of epidemiology and population health","volume":"73 2","pages":"202990"},"PeriodicalIF":0.0000,"publicationDate":"2025-04-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of epidemiology and population health","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1016/j.jeph.2025.202990","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
Mental health disorders have a high disability-adjusted life years in the Middle East and North Africa. This rise has led to a surge in related publications, prompting researchers to use AI tools like ChatGPT to reduce time spent on routine tasks. Our study aimed to validate an AI-assisted critical appraisal (CA) tool by comparing it with human raters. We developed customized GPT models using ChatGPT-4. These models were tailored to evaluate studies using the Newcastle-Ottawa Scale (NOS) or the Jadad Scale in one model, while another model evaluated STROBE or CONSORT guidelines. Our results showed a moderate to good agreement between human CA and our GPTs for the NOS for cohort, case control and cross-sectional studies and for the Jadad scale, with an ICC of 0.68 [95 %CI: 0.24-0.82], 0.69 [95 %CI: 0.31-0.88], 0.76 [95 %CI: 0.47-0.90] and 0.84 [95 %CI: 0.57-0.94] respectively. There was also a moderate to substantial agreement between the two methods for STROBE in cross sectional, cohort, case control studies, and for CONSORT in trial design, with a K of 0.63 [95 %CI: 0.56-0.70], 0.57 [95 %CI: 0.47-0.66], 0.48 [95 %CI: 0.38-0.50] and 0.70 [95 %CI: 0.63-0.77] respectively. Our custom GPT models produced hallucinations in 6.5 % and 4.9 % of cases, respectively. Human raters took an average of 19.6 ± 4.3 min per article, whereas our customized GPTs took only 1.4. ChatGPT could be a useful tool for handling repetitive tasks yet its effective application relies on the critical expertise of researchers.