ITEMS Corner Update: The Final Three Steps in the Development Process

Throughout 2023, I have detailed each step of the module development process for the Instructional Topics in Educational Measurement Series (ITEMS). In the first issue, I outlined the 10 steps necessary to complete a module. In the second issue, I detailed Steps 1–3, which cover outlining the content, developing the content in premade PowerPoint templates, and having the slides reviewed by the editor. In the third issue of the year, I outlined Step 4—recording the audio, Step 5—having the editor polish the module (e.g., animating the content), Step 6—building the activity, and Step 7—building interactive learning checks (i.e., selected response questions designed to check for understanding). In this issue, I elaborate on the final three steps: Step 8—external review, Step 9—building the module on the portal, and Step 10—writing the piece to be published in Educational Measurement: Issues and Practice (EM:IP). Following the in-depth explanation of each of these steps, I then introduce the newest module published to the ITEMS portal (https://www.ncme.org/ITEMSportal).

Authors may opt to have their module externally reviewed (Step 8) prior to recording audio (Step 4) or after the module has been polished (Step 5). Having the module content reviewed prior to recording audio allows for modifying content easily without having to do “double” work (e.g., rerecording audio on slides, reorganizing flow charts). However, many authors find that their bulleted notes for each slide are not sufficient for reviewers to understand the final product. Alternatively, they may opt to have their module sent out for review once it has been editorially polished. This lets reviewers watch the author's “final” product. Because the reviewers may suggest updates, I request authors record audio on each slide. Should an author choose to make a change after review, they then do not have to rerecord an entire 20-minute section of audio. Reviewers are instructed to provide constructive feedback and are given insights about the full process that authors have already worked through (i.e., the ten-step process). It is emphasized that the purpose of ITEMS is not to present novel cutting-edge research. Rather, it is a publication designed to provide instructional resources on current practices in the field.

After receiving reviewer feedback, authors are provided an opportunity to revise their module. Similar to a manuscript revise and resubmission, authors are asked to respond to each reviewer's comment, articulating how they have addressed each. This serves an additional purpose; specifically, this assists the editor in repolishing the updated module. For example, if audio is rerecorded on a slide, the editor will need to adjust animations and timing. After the editor has made final updates, the author reviews the module to give final approval. Upon receiving approval, the editor then builds the module onto the NCME website ITEMS portal (www.ncme.org/ITEMSportal).

Each module looks similar on the ITEMS portal. At the top of the page is the title. Beneath the title is a text abstract and description of the author(s). Below this is an introduction to the module, which is comparable to a video abstract. The sections of the module are laid out sequentially with a header, section learning objectives, the video content, a downloadable Learning Check interactive slide deck, and associated files/links. The videos are downloadable and may be embedded in other websites or learning management systems. Because each video can be downloaded, the module reference is located on the last slide embedded in the video file. At the bottom of each module webpage is a link for all downloadable content. This includes a PDF version of the slides, a reference list, and other files (e.g., datasets). Finally, there is a link to the EM:IP publication associated with the module.

The final step in the process is for the author to write the EM:IP piece. This approximately two-page manuscript presents an overview of the module to provide readers of EM:IP with enough information about the module to draw interest and scope of content. In addition to the text abstract, the module learning objectives are presented with a brief description of each section and the intended audience of the module. The manuscript undergoes the typical EM:IP publishing process, with authors reviewing a proof as well as the manuscript being assigned a DOI/official EM:IP reference that can be cited. Although the ITEMS module may be published at any time on the NCME website, the EM:IP submissions are published quarterly.

Finally, I am excited to announce Digital Module 34 by Mairead Shaw and Dr. Jessica Flake. In this module, Shaw and Flake review two different frameworks for multilevel measurement modeling: (1) multilevel modeling and 2) structural equation modeling. Throughout their module, they demonstrate the process with R code, providing the code and data so learners can follow along.

If you are interested in learning more about the ITEMS module development process, authoring a module, or being involved in some other capacity, please reach out to me at [email protected].