Identifying Factors that Increase False-Positive Rates on Embedded Performance Validity Testing in ADHD Evaluations.

Abstract

Objective: This study investigated why certain embedded performance validity indicators (EVIs) are prone to higher false-positive rates (FPRs) in attention-deficit/hyperactivity disorder (ADHD) evaluations. The first aim was to establish the relationship between FPRs and 15 EVIs derived from six cognitive tests when used independently and together among adults with ADHD who have valid test performance. The second aim was to determine which specific EVIs increase the FPRs in this population.

Method: Participants were 517 adult ADHD referrals with valid neurocognitive test performance as determined by multiple performance validity tests and established empirical criteria. FPRs were defined by the proportion of participants who scored below an empirically established EVI cutoff with ≥0.90 specificity.

Results: EVIs derived from two of the six tests exhibited unacceptably high FPRs (>10%) when used independently, but the total FPR decreased to 8.1% when the EVIs were aggregated. Several EVIs within a sustained attention test were associated with FPRs around 11%. EVIs that did not include demographically adjusted cutoffs, specifically for race, were associated with higher FPRs around 14%. Conversely, FPRs did not significantly differ based on whether EVIs included timed versus untimed, verbal versus nonverbal, or graphomotor versus non-graphomotor components, nor whether they had raw versus standardized cut scores.

Conclusions: Findings suggest that practitioners should consider both the type of test from which an EVI is derived and the aggregate number of EVIs employed to minimize the FPRs in ADHD evaluations. Findings also indicate that more nuanced approaches to validity test selection and development are needed.