Types of problems encountered by automated tool accessibility assessments, expert inspections and user testing: a systematic literature mapping

The use of websites has become essential for the development of daily activities. However, not all people can have full access to web-based services and content due to many websites being inaccessible to disabled people, such as people with visual disabilities. In this context, even though developers may demonstrate effort to attempt to create more accessible content, there is limited information about the characteristics of different web accessibility evaluation methods. Thus, the aim of the present study was to perform a meta-analysis of 38 types of accessibility problems on websites extracted from 19 studies in the literature. The studies performed automated assessments using tools, expert-based inspections and user tests involving disabled people. The results confirm other considerations made in the literature, showing that automated assessment methods have significant limitations in the adequate coverage of accessibility problems, covering less than 40% of the types of problems encountered. This way, the paper concludes by showing a consolidation of data from the literature to reinforce claims that effective web accessibility evaluations should count on expert-based inspections and user evaluations involving disabled users.