The effect of the construction of mixed anchor items on the test equating in mixed-format tests

Abstract

With the introduction of innovative items in the computer-based NAEA, the item format became diversified, and the current method of selecting anchor items from multiple-choice (MC) items became burdensome for test developers. The purpose of this study is to examine whether a mixed-format anchor items including both MC and constructed-response (CR) items could hold the stability of equating in the NAEA, and give desirable direction to expand the range of anchor items and to make test construct flexible. A simulation study was conducted based on a common-item non-equivalent group design with the test conditions of the Korean subject in the NAEA. The treatment conditions include (1) a type of anchor item set (a single MC or a mixed), (2) a composition type of mixed anchor item (MC:CR = 1:1, 3:2, or 3:4), (3) a length of mixed anchor items within a test (25%, 22%, or 19%), and (4) the presence of multidimensionality according to the item format were assumed. The main result is that there was no significant difference in the equating error between the single MC and the mixed anchor item condition. Specifically the equating error of the single MC was smaller than that of the mixed-format cases where the ratios between MC and CR were 1:1 and 3:2. In the 3:4-ratio mixed condition, the equating error was smaller than that of the single MC condition. A length of mixed anchor items within a test showed somewhat mixed results, probably because there was not much variation between the conditions in the composition ratio, about 20% in test. When there was multidimensionality according to the item format in the test, the equating error increased significantly with mixed anchor items compared to that of unidimensional test condition. In sum, about 20% of the mixed anchor items with slightly higher CR ratio was found to be suitable in the unidimensional test and caution was called for in using mixed anchor items in multidimensional test.