{"title":"Assessing the Psychometric Properties of Quality Experience in Undergraduate Research Using Item Response Theory","authors":"Tien-Ling Hu, Dubravka Svetina Valdivia","doi":"10.1007/s11162-024-09814-6","DOIUrl":null,"url":null,"abstract":"<p>Undergraduate research, recognized as one of the High-Impact Practices (HIPs), has demonstrated a positive association with diverse student learning outcomes. Understanding the pivotal quality factors essential for its efficacy is important for enhancing student success. This study evaluates the psychometric properties of survey items employed to gauge the quality of undergraduate research, including alignment with Kuh and O’Donnell’s (2013) eight HIP characteristics, alongside assessments of reliability, validity, and generalizability across demographic groups. The study assesses the validity and reliability of these measures at both the scale and item levels using data from the National Survey of Student Engagement’s (NSSE) HIP Quality Topical Module. The methodological approaches employed include Exploratory Factor Analysis, Parallel Analysis, Item Response Theory, and Differential Item Functioning (DIF). Our findings uncover a misalignment between NSSE’s HIP Quality module items and HIP characteristics, leading to the identification of seven subscales instead of eight. Nevertheless, four subscales—Reflective and Integrative Learning, Real-World Applications, Interactions with Others, and High-Performance Expectations—emerge as valid indicators of undergraduate research experiences. While specific items yield valuable insights at the item level, refinement is recommended for others. Despite the identification of two items exhibiting DIF, their negligible effect sizes suggest that major revisions are unwarranted solely on DIF grounds. This study offers recommendations for item refinement, including the incorporation of new items, wording updates, and tailored utilization of assessment tools within educational institutions. These recommendations are intended to empower educators and researchers to effectively capture the quality dimensions of students’ undergraduate research experiences, thereby fostering their academic success.</p>","PeriodicalId":48200,"journal":{"name":"Research in Higher Education","volume":"56 1","pages":""},"PeriodicalIF":1.9000,"publicationDate":"2024-09-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Research in Higher Education","FirstCategoryId":"95","ListUrlMain":"https://doi.org/10.1007/s11162-024-09814-6","RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}
引用次数: 0
Abstract
Undergraduate research, recognized as one of the High-Impact Practices (HIPs), has demonstrated a positive association with diverse student learning outcomes. Understanding the pivotal quality factors essential for its efficacy is important for enhancing student success. This study evaluates the psychometric properties of survey items employed to gauge the quality of undergraduate research, including alignment with Kuh and O’Donnell’s (2013) eight HIP characteristics, alongside assessments of reliability, validity, and generalizability across demographic groups. The study assesses the validity and reliability of these measures at both the scale and item levels using data from the National Survey of Student Engagement’s (NSSE) HIP Quality Topical Module. The methodological approaches employed include Exploratory Factor Analysis, Parallel Analysis, Item Response Theory, and Differential Item Functioning (DIF). Our findings uncover a misalignment between NSSE’s HIP Quality module items and HIP characteristics, leading to the identification of seven subscales instead of eight. Nevertheless, four subscales—Reflective and Integrative Learning, Real-World Applications, Interactions with Others, and High-Performance Expectations—emerge as valid indicators of undergraduate research experiences. While specific items yield valuable insights at the item level, refinement is recommended for others. Despite the identification of two items exhibiting DIF, their negligible effect sizes suggest that major revisions are unwarranted solely on DIF grounds. This study offers recommendations for item refinement, including the incorporation of new items, wording updates, and tailored utilization of assessment tools within educational institutions. These recommendations are intended to empower educators and researchers to effectively capture the quality dimensions of students’ undergraduate research experiences, thereby fostering their academic success.
期刊介绍:
Research in Higher Education publishes studies that examine issues pertaining to postsecondary education. The journal is open to studies using a wide range of methods, but has particular interest in studies that apply advanced quantitative research methods to issues in postsecondary education or address postsecondary education policy issues. Among the topics of interest to the journal are: access and retention; student success; equity; faculty issues; institutional productivity and assessment; postsecondary education governance; curriculum and instruction; state and federal higher education policy; and financing of postsecondary education. The journal encourages submissions from scholars in disciplines outside of higher education, and studies from outside the United States that address issues that are of interest to the readership. The journal will on occasion publish short notes of a methodological nature, literature reviews of topics pertaining to postsecondary research, and “research and practice” studies illustrating how postsecondary research can inform decision making.