Rasi Mizori, Muhayman Sadiq, Malik Takreem Ahmad, Anthony Siu, Reubeen Rashid Ahmad, Zijing Yang, Helen Oram, James Galloway
{"title":"STEM 考试成绩：大语言模型时代的开卷与闭卷考试方法。","authors":"Rasi Mizori, Muhayman Sadiq, Malik Takreem Ahmad, Anthony Siu, Reubeen Rashid Ahmad, Zijing Yang, Helen Oram, James Galloway","doi":"10.1111/tct.13839","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Background</h3>\n \n <p>The COVID-19 pandemic accelerated the shift to remote learning, heightening scrutiny of open-book examinations (OBEs) versus closed-book examinations (CBEs) within science, technology, engineering, arts and mathematics (STEM) education. This study evaluates the efficacy of OBEs compared to CBEs on student performance and perceptions within STEM subjects, considering the emerging influence of sophisticated large language models (LLMs) such as GPT-3.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>Adhering to PRISMA guidelines, this systematic review analysed peer-reviewed articles published from 2013, focusing on the impact of OBEs and CBEs on university STEM students. Standardised mean differences were assessed using a random effects model, with heterogeneity evaluated by <i>I</i><sup>2</sup> statistics, Cochrane's <i>Q</i> test and Tau statistics.</p>\n </section>\n \n <section>\n \n <h3> Results</h3>\n \n <p>Analysis of eight studies revealed mixed outcomes. Meta-analysis showed that OBEs generally resulted in better scores than CBEs, despite significant heterogeneity (<i>I</i><sup>2</sup> = 97%). Observational studies displayed more pronounced effects, with noted concerns over technical difficulties and instances of cheating.</p>\n </section>\n \n <section>\n \n <h3> Discussion</h3>\n \n <p>Results suggest that OBEs assess competencies more aligned with current educational paradigms than CBEs. However, the emergence of LLMs poses new challenges to OBE validity by simplifying the generation of comprehensive answers, impacting academic integrity and examination fairness.</p>\n </section>\n \n <section>\n \n <h3> Conclusions</h3>\n \n <p>While OBEs are better suited to contemporary educational needs, the influence of LLMs on their effectiveness necessitates further study. Institutions should prudently consider the competencies assessed by OBEs, particularly in light of evolving technological landscapes. Future research should explore the integrity of OBEs in the presence of LLMs to ensure fair and effective student evaluations.</p>\n </section>\n </div>","PeriodicalId":47324,"journal":{"name":"Clinical Teacher","volume":"22 1","pages":""},"PeriodicalIF":1.4000,"publicationDate":"2024-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11663729/pdf/","citationCount":"0","resultStr":"{\"title\":\"STEM exam performance: Open- versus closed-book methods in the large language model era\",\"authors\":\"Rasi Mizori, Muhayman Sadiq, Malik Takreem Ahmad, Anthony Siu, Reubeen Rashid Ahmad, Zijing Yang, Helen Oram, James Galloway\",\"doi\":\"10.1111/tct.13839\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div>\\n \\n \\n <section>\\n \\n <h3> Background</h3>\\n \\n <p>The COVID-19 pandemic accelerated the shift to remote learning, heightening scrutiny of open-book examinations (OBEs) versus closed-book examinations (CBEs) within science, technology, engineering, arts and mathematics (STEM) education. This study evaluates the efficacy of OBEs compared to CBEs on student performance and perceptions within STEM subjects, considering the emerging influence of sophisticated large language models (LLMs) such as GPT-3.</p>\\n </section>\\n \\n <section>\\n \\n <h3> Methods</h3>\\n \\n <p>Adhering to PRISMA guidelines, this systematic review analysed peer-reviewed articles published from 2013, focusing on the impact of OBEs and CBEs on university STEM students. Standardised mean differences were assessed using a random effects model, with heterogeneity evaluated by <i>I</i><sup>2</sup> statistics, Cochrane's <i>Q</i> test and Tau statistics.</p>\\n </section>\\n \\n <section>\\n \\n <h3> Results</h3>\\n \\n <p>Analysis of eight studies revealed mixed outcomes. Meta-analysis showed that OBEs generally resulted in better scores than CBEs, despite significant heterogeneity (<i>I</i><sup>2</sup> = 97%). Observational studies displayed more pronounced effects, with noted concerns over technical difficulties and instances of cheating.</p>\\n </section>\\n \\n <section>\\n \\n <h3> Discussion</h3>\\n \\n <p>Results suggest that OBEs assess competencies more aligned with current educational paradigms than CBEs. However, the emergence of LLMs poses new challenges to OBE validity by simplifying the generation of comprehensive answers, impacting academic integrity and examination fairness.</p>\\n </section>\\n \\n <section>\\n \\n <h3> Conclusions</h3>\\n \\n <p>While OBEs are better suited to contemporary educational needs, the influence of LLMs on their effectiveness necessitates further study. Institutions should prudently consider the competencies assessed by OBEs, particularly in light of evolving technological landscapes. Future research should explore the integrity of OBEs in the presence of LLMs to ensure fair and effective student evaluations.</p>\\n </section>\\n </div>\",\"PeriodicalId\":47324,\"journal\":{\"name\":\"Clinical Teacher\",\"volume\":\"22 1\",\"pages\":\"\"},\"PeriodicalIF\":1.4000,\"publicationDate\":\"2024-11-04\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11663729/pdf/\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Clinical Teacher\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://onlinelibrary.wiley.com/doi/10.1111/tct.13839\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q4\",\"JCRName\":\"MEDICINE, RESEARCH & EXPERIMENTAL\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Clinical Teacher","FirstCategoryId":"1085","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1111/tct.13839","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q4","JCRName":"MEDICINE, RESEARCH & EXPERIMENTAL","Score":null,"Total":0}
STEM exam performance: Open- versus closed-book methods in the large language model era
Background
The COVID-19 pandemic accelerated the shift to remote learning, heightening scrutiny of open-book examinations (OBEs) versus closed-book examinations (CBEs) within science, technology, engineering, arts and mathematics (STEM) education. This study evaluates the efficacy of OBEs compared to CBEs on student performance and perceptions within STEM subjects, considering the emerging influence of sophisticated large language models (LLMs) such as GPT-3.
Methods
Adhering to PRISMA guidelines, this systematic review analysed peer-reviewed articles published from 2013, focusing on the impact of OBEs and CBEs on university STEM students. Standardised mean differences were assessed using a random effects model, with heterogeneity evaluated by I2 statistics, Cochrane's Q test and Tau statistics.
Results
Analysis of eight studies revealed mixed outcomes. Meta-analysis showed that OBEs generally resulted in better scores than CBEs, despite significant heterogeneity (I2 = 97%). Observational studies displayed more pronounced effects, with noted concerns over technical difficulties and instances of cheating.
Discussion
Results suggest that OBEs assess competencies more aligned with current educational paradigms than CBEs. However, the emergence of LLMs poses new challenges to OBE validity by simplifying the generation of comprehensive answers, impacting academic integrity and examination fairness.
Conclusions
While OBEs are better suited to contemporary educational needs, the influence of LLMs on their effectiveness necessitates further study. Institutions should prudently consider the competencies assessed by OBEs, particularly in light of evolving technological landscapes. Future research should explore the integrity of OBEs in the presence of LLMs to ensure fair and effective student evaluations.
期刊介绍：
The Clinical Teacher has been designed with the active, practising clinician in mind. It aims to provide a digest of current research, practice and thinking in medical education presented in a readable, stimulating and practical style. The journal includes sections for reviews of the literature relating to clinical teaching bringing authoritative views on the latest thinking about modern teaching. There are also sections on specific teaching approaches, a digest of the latest research published in Medical Education and other teaching journals, reports of initiatives and advances in thinking and practical teaching from around the world, and expert community and discussion on challenging and controversial issues in today"s clinical education.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
