Inter-rater Reliability of Command-Line Web Accessibility Evaluation Tools

Abstract

This study compares four command-line interface (CLI) web accessibility tools, examining if one CLI tool is sufficient for automated accessibility evaluation. The four tools were: Axe-core/cli, IBM Equal Access NPM Accessibility Checker (Accessibility Checker), Pa11y-ci, and the A11y Machine. Inter-rater reliability was calculated using Gwet’s alpha coefficient 2 (AC2), and the results indicate very poor reliability between tools.