Design, development, and evaluation of the organic chemistry representational competence assessment (ORCA)†

Abstract

This paper describes the design and evaluation of the rganic chemistry epresentational ompetence ssessment (ORCA). Grounded in Kozma and Russell's representational competence framework, the ORCA measures the learner's ability to interpret, translate, and use six commonly used representations of molecular structure (condensed structures, Lewis structures, skeletal structures, wedge-dash diagrams, Newman projections, and chair conformations). Semi-structured interviews with 38 first-semester organic chemistry learners informed the development of the ORCA items. The ORCA was developed and refined through three pilot administrations involving a total of 3477 first-semester organic chemistry students from multiple institutions. The final version of the ORCA was completed by 1494 students across five institutions. Various analyses provided evidence for the validity and reliability of the data generated by the assessment. Both one-factor and three-factor correlated structures were explored via confirmatory factor analysis. The one-factor model better captured the underlying structure of the data, which suggests that representational competence is better evaluated as a unified construct rather than as distinct, separate skills. The ORCA data reveal that the representational competence skills are interconnected and should consistently be reinforced throughout the organic chemistry course.