Assessing the Psychometric Properties of Quality Experience in Undergraduate Research Using Item Response Theory

IF 1.9 3区 教育学 Q2 EDUCATION & EDUCATIONAL RESEARCH
Tien-Ling Hu, Dubravka Svetina Valdivia
{"title":"Assessing the Psychometric Properties of Quality Experience in Undergraduate Research Using Item Response Theory","authors":"Tien-Ling Hu, Dubravka Svetina Valdivia","doi":"10.1007/s11162-024-09814-6","DOIUrl":null,"url":null,"abstract":"<p>Undergraduate research, recognized as one of the High-Impact Practices (HIPs), has demonstrated a positive association with diverse student learning outcomes. Understanding the pivotal quality factors essential for its efficacy is important for enhancing student success. This study evaluates the psychometric properties of survey items employed to gauge the quality of undergraduate research, including alignment with Kuh and O’Donnell’s (2013) eight HIP characteristics, alongside assessments of reliability, validity, and generalizability across demographic groups. The study assesses the validity and reliability of these measures at both the scale and item levels using data from the National Survey of Student Engagement’s (NSSE) HIP Quality Topical Module. The methodological approaches employed include Exploratory Factor Analysis, Parallel Analysis, Item Response Theory, and Differential Item Functioning (DIF). Our findings uncover a misalignment between NSSE’s HIP Quality module items and HIP characteristics, leading to the identification of seven subscales instead of eight. Nevertheless, four subscales—Reflective and Integrative Learning, Real-World Applications, Interactions with Others, and High-Performance Expectations—emerge as valid indicators of undergraduate research experiences. While specific items yield valuable insights at the item level, refinement is recommended for others. Despite the identification of two items exhibiting DIF, their negligible effect sizes suggest that major revisions are unwarranted solely on DIF grounds. This study offers recommendations for item refinement, including the incorporation of new items, wording updates, and tailored utilization of assessment tools within educational institutions. These recommendations are intended to empower educators and researchers to effectively capture the quality dimensions of students’ undergraduate research experiences, thereby fostering their academic success.</p>","PeriodicalId":48200,"journal":{"name":"Research in Higher Education","volume":"56 1","pages":""},"PeriodicalIF":1.9000,"publicationDate":"2024-09-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Research in Higher Education","FirstCategoryId":"95","ListUrlMain":"https://doi.org/10.1007/s11162-024-09814-6","RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}
引用次数: 0

Abstract

Undergraduate research, recognized as one of the High-Impact Practices (HIPs), has demonstrated a positive association with diverse student learning outcomes. Understanding the pivotal quality factors essential for its efficacy is important for enhancing student success. This study evaluates the psychometric properties of survey items employed to gauge the quality of undergraduate research, including alignment with Kuh and O’Donnell’s (2013) eight HIP characteristics, alongside assessments of reliability, validity, and generalizability across demographic groups. The study assesses the validity and reliability of these measures at both the scale and item levels using data from the National Survey of Student Engagement’s (NSSE) HIP Quality Topical Module. The methodological approaches employed include Exploratory Factor Analysis, Parallel Analysis, Item Response Theory, and Differential Item Functioning (DIF). Our findings uncover a misalignment between NSSE’s HIP Quality module items and HIP characteristics, leading to the identification of seven subscales instead of eight. Nevertheless, four subscales—Reflective and Integrative Learning, Real-World Applications, Interactions with Others, and High-Performance Expectations—emerge as valid indicators of undergraduate research experiences. While specific items yield valuable insights at the item level, refinement is recommended for others. Despite the identification of two items exhibiting DIF, their negligible effect sizes suggest that major revisions are unwarranted solely on DIF grounds. This study offers recommendations for item refinement, including the incorporation of new items, wording updates, and tailored utilization of assessment tools within educational institutions. These recommendations are intended to empower educators and researchers to effectively capture the quality dimensions of students’ undergraduate research experiences, thereby fostering their academic success.

Abstract Image

用项目反应理论评估本科生科研质量体验的心理测量特性
本科生研究是公认的 "高效实践"(HIPs)之一,它与学生的不同学习成果有着积极的联系。了解对其有效性至关重要的关键质量因素对于提高学生的成功率非常重要。本研究评估了用于衡量本科生研究质量的调查项目的心理测量特性,包括与 Kuh 和 O'Donnell(2013 年)提出的八个 HIP 特性的一致性,以及对可靠性、有效性和跨人口群体普适性的评估。本研究利用全国学生参与度调查(NSSE)HIP 质量主题模块的数据,从量表和项目两个层面评估了这些测量方法的有效性和可靠性。采用的方法包括探索性因子分析、平行分析、项目反应理论和差异项目功能(DIF)。我们的研究结果发现,NSSE 的 HIP 质量模块项目与 HIP 特征之间存在不一致,导致确定了七个分量表而不是八个分量表。尽管如此,四个子量表--反思性和综合性学习、真实世界应用、与他人互动和高性能期望--仍然是本科生研究经历的有效指标。虽然某些项目在项目层面上提出了有价值的见解,但建议对其他项目进行改进。尽管发现了两个表现出 DIF 的项目,但它们的效应大小可以忽略不计,这表明仅以 DIF 为由进行重大修订是没有必要的。本研究提出了改进项目的建议,包括纳入新项目、更新措辞以及在教育机构内有针对性地使用测评工具。这些建议旨在增强教育工作者和研究人员的能力,以有效捕捉学生本科研究经历的质量维度,从而促进他们的学业成功。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Research in Higher Education
Research in Higher Education EDUCATION & EDUCATIONAL RESEARCH-
CiteScore
4.40
自引率
9.50%
发文量
52
期刊介绍: Research in Higher Education publishes studies that examine issues pertaining to postsecondary education. The journal is open to studies using a wide range of methods, but has particular interest in studies that apply advanced quantitative research methods to issues in postsecondary education or address postsecondary education policy issues. Among the topics of interest to the journal are: access and retention; student success; equity; faculty issues; institutional productivity and assessment; postsecondary education governance; curriculum and instruction; state and federal higher education policy; and financing of postsecondary education. The journal encourages submissions from scholars in disciplines outside of higher education, and studies from outside the United States that address issues that are of interest to the readership. The journal will on occasion publish short notes of a methodological nature, literature reviews of topics pertaining to postsecondary research, and “research and practice” studies illustrating how postsecondary research can inform decision making.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信