Comparing the NHTSA and ISO occlusion test protocols: how many participants are sufficient?

Occlusion is a popular technique used to evaluate the visual demand associated with secondary tasks/devices in a driving context offering a low cost, highly accessible alternative to driving simulators and on-road studies. Several standardised occlusion test protocols have been published, most notably by NHTSA and ISO. These differ significantly in terms of how many participants are deemed to be sufficient in order to elicit statistically representative behaviour, and thus impose different 'costs' on incumbent organisations. A NHTSA-compliant study investigated three navigation-related tasks using a smartphone app (APP) and portable navigation device (PND). As a comparison, 1000 iterations of 10 participant ISO sample groups were extracted from the NHTSA 24-participant cohort and analysed in accordance with the same measures. Results obtained using all 10-participant ISO groups were consistent with the NHTSA findings, indicating that both standards would support the same conclusions regarding relative performance. Applying NHTSA's acceptance criteria, it is evident that, in most cases, recommendations based on the smaller ISO samples would be the same as those obtained from the full NHTSA cohort. However, only 95.5% conformance was observed for task 3 using the PND, suggesting a small risk that different conclusions could be drawn for this particular device/task if a smaller number of participants were used. Given the lower inherent costs associated with the ISO protocol, we thus conclude that this may be better suited for relative/formative assessment, whereas the NHTSA protocol may be more appropriate for summative evaluations.