Information Retrieval Techniques for Automated Policy Review

Abstract

In this paper we adapt standard information retrieval techniques to a novel task, the mandatory regulatory review of public comments on proposed rule changes. The vast number of public comments exceeds the responsible agency’s ability to manually review in the time allowed. Therefore, the agency requires an automated approach to efficiently sort and process the comments. To rank the public comments’ relevance to rule sections, we implement a vector space model and compare the results to experts’ reviews. We perform experiments over several indexing techniques to improve semantic relevance, splitting the regulatory document based on textual formatting, text length, and a hybrid method combining these two techniques. To improve the accuracy of our predictions, we test various synonym lists generated from a domain-specific ontology, as well as variations of standard stopword lists. By applying the relevance search as a multi-class classification problem, we find the method that most closely matches human reviews, achieving respective normalized discounted cumulative gain and mean average precision scores of 0.83 and 0.75 on our test data set.