Supporting fair use in context of instruments in engineering education research: A renewed conversation about validity

Abstract

London et al. (2021) described engineering education as a “maturing field of scientific inquiry” that lacks the “signposts,” such as theories, techniques, instruments, and methods, characteristic of more mature fields. It is perhaps for this reason that engineering education research has historically relied on instruments developed by scholars outside of engineering education (e.g., psychology, sociology, and science education; Borrego & Streveler, 2015; Malmi et al., 2018). This history has created a body of extant literature with varying levels of consideration made to the unifying concept of validity and the process of validation as an ongoing argument for the fair use of an instrument with the population under study and in the context in which the study occurs. The sheer quantity of studies having overlooked such considerations can easily suggest to new scholars that the approaches of yesterday are still appropriate for today's engineering education research.

Douglas and Purzer (2015) brought to the forefront how validity was conceptualized in their editorial written a decade ago. They highlighted the 2014 Standards for Educational and Psychological Testing by the American Psychological Association, American Educational Research Association, and the National Council for Measurement in Education, presented five misconceptions about validity, suggested clarifications to the community to improve assessment-based research, and shared several exemplar assessment instruments for the community to consider. New examples of exemplar assessment instruments have been published in engineering education since, but patterns in publications and submissions wherein authors fail to provide a robust description of the validation process persist. This pattern calls into questions instrument validity, trust in the associated findings, and the ability of scholars to adopt or adapt an instrument for fair use in context for subsequent studies.

The discussion for this editorial emerged from experiences across the Editorial Board of the Journal of Engineering Education (JEE). Many submissions continue to omit or overlook modern definitions of validity; in particular, they fail to engage in validation as both a process and an argument. The result is major revisions or rejections for otherwise promising submissions. As an author team, we represent different faculty ranks and roles within JEE, including assistant professor and Assistant Editor, Trevion Henderson; associate professor and Associate Editor, Allison Godwin; and professor, department chair, and Deputy Editor, Adam Carberry. Each of us has experience with instrument development, the editorial process for shepherding manuscripts through the review process, and publishing our own work in this space. We acknowledge that the discussion provided reflects our positionality as US-based scholars and that there are other framings of validity. Our goal for this editorial is to highlight some of the expectations for the publication of instrument development manuscripts in engineering education to better serve the research community and improve the review process for this type of work.

Since Douglas and Purzer's (2015) editorial, the definition of validity and standards for evidence in instrument development work have changed. Of particular importance is the definition and standard for validity. The previous standards were a trinitarian view with three distinct facets: (1) content (i.e., Does the instrument measure the domain it is designed to measure?), (2) construct (i.e., Does the instrument accurately reflect the theoretical concept it is intended to measure?), and (3) criterion validity (i.e., Does the instrument predict a real-world outcome that it is designed to measure?; Cronbach & Meehl, 1955). The shift has been to a unitarian view of validity in which multiple sources of evidence are used to support an argument for fair use of the instrument with the population under study and within the context of a particular study (Messick, 1995). This new framing purports that there is not one process for establishing validity, and instruments do not possess the static property of being “valid” or “validated.” Each instance of instrument use in engineering education research should be accompanied by validity evidence for the measures used, which provides a foundational argument that allows consumers of research to better gauge the trustworthiness of the work being presented.

Establishing validity is a matter of significant scholarly discussion and debate. For the purposes of discussion, we describe a general framework for building an argument for validity in instrument development manuscripts in engineering education that should not be viewed as prescriptive or an exhaustive list necessary for publication. Instead, we implore authors to integrate the issues noted to help ensure that papers have appropriate validity evidence.

We argue that alignment of the issues discussed supports fair use in future research, as well as reproducibility across contexts. We encourage the engineering education community to engage with this work as it provides a valuable discussion of validity that has not yet been taken up broadly within the field of engineering education. To assist, we provide a table of reflective questions with example references in the Appendix for the community to engage with in the production of scholarship related to measurement and the use of instruments.