{"title":"The Impact of Differential Item Functioning on the Warwick-Edinburgh Mental Well-Being Scale.","authors":"Hong Eng Goh, Ida Marais, Michael Ireland","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>Establishing the internal validity of psychometric instruments is an important research priority, and is especially vital for instruments that are used to collect data to guide public policy decisions. The Warwick-Edinburgh Mental Well-Being Scale (WEMWBS) is a well-established and widely-used instrument for assessing individual differences in well-being. The current analyses were motivated by concerns that metal wellbeing items that refer to interpersonal relationships (Items 9 and 12) may operate differently for those in a relationship compared to those not in a relationship. To assess this, the present study used item characteristic curves (ICC) and ANOVA of residuals to scrutinize the differential item functioning (DIF) of the 14 WEMWBS items for participant relationship status (n with partner = 261, n without partner = 210). Items 5, 9, and 12 showed evidence of DIF which impacted group mean differences. Item 5 (\"energy to spare\") was unexpected, however plausible explanation is discussed. For participants at the same level of mental wellbeing, those in a relationship scored higher on items 9 and 12 than those not in a relationship. This suggests these items are sensitive to non-wellbeing related variance associated with relationship status. Implications and future research directions are discussed.</p>","PeriodicalId":73608,"journal":{"name":"Journal of applied measurement","volume":"19 2","pages":"162-172"},"PeriodicalIF":0.0,"publicationDate":"2018-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"36216477","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ruth Chu-Lien Chao, Kathy Green, Kranti Dugar, Joseph Longo
{"title":"Rasch Analysis of the Brief Symptom Inventory-18 with African Americans.","authors":"Ruth Chu-Lien Chao, Kathy Green, Kranti Dugar, Joseph Longo","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>Although the United States offers some of the most advanced psychological services in the world, not everyone in U.S. shares equally in these services, and health disparities persist when assessments do not appropriately measure different populations' mental health problems. To address this assessment issue, we conducted factor and Rasch analyses to assess the psychometric characteristics of the Brief Symptom Inventory-18 (BSI-18) to evaluate whether the BSI is culturally appropriate for assessing African Americans' psychological distress. The dimensional structure of the BSI was first identified and held up under cross-validation with a second subsample. The measure was unidimensional among African Americans. Our results also suggested minimal person separation, stability across subsamples, and little differential item functioning. Most African Americans identified themselves on the low end of the categories in a 0-4 rating scale, indicating their low endorsement of the items on the BSI. Rasch analyses were completed with the original scale but also collapsing the scale to three points, with some increase in separation and reliability for the collapsed scale. Differences in mean person position were found for mental health-related variables, consistent with hypotheses. Implications for theory and research on multicultural health scales are discussed as are effects of severe item skewness on analyses.</p>","PeriodicalId":73608,"journal":{"name":"Journal of applied measurement","volume":"19 2","pages":"173-191"},"PeriodicalIF":0.0,"publicationDate":"2018-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"36216478","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A Facets Analysis of Analytic vs. Holistic Scoring of Identical Short Constructed-Response Items: Different Outcomes and Their Implications for Scoring Rubric Development.","authors":"Milja Curcin, Ezekiel Sweiry","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>In scoring short constructed-response items it may be possible to apply different rubric types depending on the trait of achievement assessed. A rating scale and a partial credit Many-Facet Rasch Models (MFRM) were used to investigate whether levels-based (holistic) and hybrid (analytic) scoring rubrics functioned interchangeably when scoring short-response English reading comprehension test items. Whereas most research in similar contexts has focused solely on rater reliability, the use of MFRM in this study enabled examination of both the reliability and rating scale functioning aspects of scoring rubrics in parallel. It also enabled consideration of their effects on item and examinee parameters. This more comprehensive approach allowed the findings to be linked to general notions of rubric construct-relevance and score interpretation, and to demonstrate an approach to generating evidence for a more balanced consideration of advantages and disadvantages of each rubric in terms of both reliability and validity.</p>","PeriodicalId":73608,"journal":{"name":"Journal of applied measurement","volume":"18 3","pages":"228-246"},"PeriodicalIF":0.0,"publicationDate":"2017-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"35950093","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"The Use of Differential Item Functioning (DIF) Analysis to Distinguish Between Similar Job Roles.","authors":"Nicole M Risk, James R Fidler","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>Two primary roles in the clinical laboratory are those of Medical Technologist (MT) and Medical Laboratory Technician (MLT). Job analyses, which form the foundation of test blueprints employed for credentialing practitioners, suggest a reasonable amount of overlap in the tasks performed by MTs and MLTs. However, credentialing assessments must clearly distinguish between the two roles and ensure that they address competencies appropriate to each practice designation. Differential item functioning (DIF) analysis techniques were applied to explore and differentiate the two laboratory practitioner job roles as an aspect of examination development. Results from the analysis suggest a high degree of similarity between these two groups in terms of scope of tasks performed. Subject matter expert interpretation suggests that the assessments are more appropriately differentiated by underlying level of task knowledge rather than scope of tasks. DIF may be applicable to other exploratory investigations that seek to differentiate job roles comprised of common competencies.</p>","PeriodicalId":73608,"journal":{"name":"Journal of applied measurement","volume":"18 2","pages":"137-150"},"PeriodicalIF":0.0,"publicationDate":"2017-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"35454044","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"General Ability or Distinct Scholastic Aptitudes? A Multidimensional Validity Analysis of a Psychometric Higher-Education Entrance Test.","authors":"Dvir Kleper, Noa Saka","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>The present study explored the construct validity of the Psychometric Entrance Test (PET) for higher education in Israel, as represented by the factorial structure of the scholastic aptitudes it measures, and focused on whether the test presents a single measure of overall ability or a measure of the fields of knowledge that are being tested. In Study 1, we used Exploratory Factor Analysis to generate hypotheses regarding the factorial structure of the test. In Study 2, Confirmatory Factor Analysis was carried out to compare competing models that were constructed based on theoretical considerations and the results of Study 1. The findings indicated that a two-layered hierarchical model, encompassing both a General Ability factor and three scholastic domain-specific factors (Verbal Reasoning, Quantitative Reasoning, and English), showed the best fit. Within the framework of the CFA, several statistical procedures were applied to assess reliability (indicator and complexity) and validity (convergent and divergent.).</p>","PeriodicalId":73608,"journal":{"name":"Journal of applied measurement","volume":"18 2","pages":"194-214"},"PeriodicalIF":0.0,"publicationDate":"2017-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"35454490","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Marcos Cupani, Tatiana Castro Zamparella, Gisella Piumatti, Grupo Vinculado
{"title":"Development of an Item Bank for the Assessment of Knowledge on Biology in Argentine University Students.","authors":"Marcos Cupani, Tatiana Castro Zamparella, Gisella Piumatti, Grupo Vinculado","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>The calibration of item banks provides the basis for computerized adaptive testing that ensures high diagnostic precision and minimizes participants' test burden. This study aims to develop a bank of items to measure the level of Knowledge on Biology using the Rasch model. The sample consisted of 1219 participants that studied in different faculties of the National University of Cordoba (mean age = 21.85 years, SD = 4.66; 66.9% are women). The items were organized in different forms and into separate subtests, with some common items across subtests. The students were told they had to answer 60 questions of knowledge on biology. Evaluation of Rasch model fit (Zstd >|2.0|), differential item functioning, dimensionality, local independence, item and person separation (>2.0), and reliability (>.80) resulted in a bank of 180 items with good psychometric properties. The bank provides items with a wide range of content coverage and may serve as a sound basis for computerized adaptive testing applications. The contribution of this work is significant in the field of educational assessment in Argentina.</p>","PeriodicalId":73608,"journal":{"name":"Journal of applied measurement","volume":"18 3","pages":"360-369"},"PeriodicalIF":0.0,"publicationDate":"2017-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"35948055","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Mapping a Data Modeling and Statistical Reasoning Learning Progression using Unidimensional and Multidimensional Item Response Models.","authors":"Robert Schwartz, Elizabeth Ayers, Mark Wilson","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>There are different ways to conceive and measure learning progressions. The approach used by the ADMSR project followed the \"four building blocks\" approach outlined by the Berkeley Evaluation and Assessment Research (BEAR) Center and the BEAR Assessment System. The final building block of this approach involves the application of a measurement model. This paper focuses on the application of unidimensional and multidimensional item response theory (IRT) measurement models to the data from the ADMSR project. Unidimensional IRT models are applied to aid in construct development and validation to see if the proposed theory of development presented by the construct map is supported by the results from an administration of the instrument. Multidimensional IRT measurement models are applied to examine the relationships between the seven constructs in the ADMSR learning progression. When applying the multidimensional model, specific links between levels of the constructs are analyzed across constructs after the application of a technique to align the seven dimensions.</p>","PeriodicalId":73608,"journal":{"name":"Journal of applied measurement","volume":"18 3","pages":"268-298"},"PeriodicalIF":0.0,"publicationDate":"2017-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"35948995","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Stefanie A Wind, Meltem Alemdar, Jeremy A Lingle, Jessica D Gale, Roxanne A Moore
{"title":"Developing an Engineering Design Process Assessment using Mixed Methods.","authors":"Stefanie A Wind, Meltem Alemdar, Jeremy A Lingle, Jessica D Gale, Roxanne A Moore","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>Recent reforms in science education worldwide include an emphasis on engineering design as a key component of student proficiency in the Science, Technology, Engineering, and Mathematics disciplines. However, relatively little attention has been directed to the development of psychometrically sound assessments for engineering. This study demonstrates the use of mixed methods to guide the development and revision of K-12 Engineering Design Process (EDP) assessment items. Using results from a middle-school EDP assessment, this study illustrates the combination of quantitative and qualitative techniques to inform item development and revisions. Overall conclusions suggest that the combination of quantitative and qualitative evidence provides an in-depth picture of item quality that can be used to inform the revision and development of EDP assessment items. Researchers and practitioners can use the methods illustrated here to gather validity evidence to support the interpretation and use of new and existing assessments.</p>","PeriodicalId":73608,"journal":{"name":"Journal of applied measurement","volume":"18 2","pages":"100-121"},"PeriodicalIF":0.0,"publicationDate":"2017-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"35454042","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Dan Cloney, Cuc Nguyen, Raymond J Adams, Collette Tayler, Gordon Cleveland, Karen Thorpe
{"title":"Psychometric Properties of the Classroom Assessment Scoring System (Pre-K): Implications for Measuring Interaction Quality in Diverse Early Childhood Settings.","authors":"Dan Cloney, Cuc Nguyen, Raymond J Adams, Collette Tayler, Gordon Cleveland, Karen Thorpe","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>The Classroom Assessment Scoring System (CLASS) is an observational instrument assessing the nature of everyday interactions in educational settings. The instrument has strong theoretical groundings; however, prior empirical validation of the CLASS has exposed some psychometric weaknesses. Further the instrument has not been the subject of psychometric analysis at the indicator level. Using a large dataset including observations of 993 Australian classrooms, confirmatory factor analysis is used to replicate findings from the few existing validation studies. Item response modelling is used to examine individual indicator behaviour. Latent growth models are used to produce new findings about estimating factor scores. Findings show that the CLASS exhibits stable psychometric properties within classrooms over repeated observations. Model fit is improved and factor scores are more reliable when the repeated-observations made in administering the CLASS are accounted for statistically. It is recommended that researchers enforce a fixed number of repeated observations to minimise bias.</p>","PeriodicalId":73608,"journal":{"name":"Journal of applied measurement","volume":"18 3","pages":"299-318"},"PeriodicalIF":0.0,"publicationDate":"2017-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"35948997","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Confidence to Perform in the Global Marketplace: Constructing and Validating a Survey Instrument for Community College Students.","authors":"Snejana Slantcheva-Durst, Mingyang Liu","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>This article discusses the construction and validation of an instrument to gauge community college students' confidence to perform in the global marketplace. The instrument was designed to capture students' beliefs in their own abilities to successfully carry out job-related tasks in cross-cultural work environments that are globally-interconnected and constantly at flux. The instrument items emerged from a comprehensive review of literature, nationwide workforce skills initiatives, rounds of expert panel analyses, and focus groups. Items were formulated within Bandura's framework of self-efficacy, and the instrument was analyzed with Rasch measurement. The Rasch analysis, conducted on a sample of 741 students, provided evidence of the content validity of the items, the generalizability of the measure, and its external validity. The instrument can offer useful feedback to community college internationalization-focused staff in their efforts to assess outcomes of international initiatives for community college students, thus supporting program assessment, evaluation of student growth, and institutional decision-making.</p>","PeriodicalId":73608,"journal":{"name":"Journal of applied measurement","volume":"18 4","pages":"434-448"},"PeriodicalIF":0.0,"publicationDate":"2017-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"35665984","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}