Are users the gold standard for accessibility evaluation?

Abstract

User testing is considered a key part of web accessibility evaluation. However, little is known about how effective is for identifying accessibility problems. Our experience, informed by a series of studies with blind users, corroborates that a website with a significant number of guideline violations can be perceived as accessible, and on the contrary, some participants may not perceive a highly accessible website as accessible. Accessibility guidelines are often criticised by their partial coverage and questionable validity. However, we should be very careful about making categorical statements in this regard as there are a number of variables that may introduce biases in user tests. We identify sources of bias related to user expertise, the experimental setting, employed language and reporting that, if not adequately controlled, may influence on the validity and reliability of the evaluation results. We discuss the limitations and practical implications of user testing with blind users for web accessibility evaluation.