The role of evaluation in AI and law: an examination of its different forms in the AI and law journal

This paper explores the presence and forms of evaluation in articles published in the journal Artificial Intelligence and Law for the ten-year period from 2005 through 2014. It represents a meta-level study of some the most significant works produced by the AI and Law community, in this case nearly 140 research articles published in the AI and Law journal. It also compares its findings to previous work conducted on evaluation appearing in the Proceedings of the International Conference on Artificial Intelligence and Law (ICAIL). In addition, the paper highlights works harnessing performance evaluation as one of their chief scientific tools and the means by which they use it. It extends the argument for why evaluation is essential in formal Artificial Intelligence and Law reports such as those in the journal. As in the case of two earlier works on the topic, it pursues answers to the questions: how good is the system, algorithm or proposal?, how reliable is the approach or technique?, and, ultimately, does the method work? The paper investigates the role of performance evaluation in scientific research reports, underscoring the argument that a performance-based 'ethic' signifies a level of maturity and scientific rigor within a community. In addition, the work examines recent publications that address the same critical issue within the broader field of Artificial Intelligence.