Semantic crawling: An approach based on Named Entity Recognition

Abstract

Law Enforcement Agencies (LEAs) are increasingly more reliant on information and communication technologies and affected by a society shaped by the Internet. The richness and quantity of information available from open sources, if properly gathered and processed, can provide valuable intelligence and help in drawing inferences from existing closed source intelligence. Today the intelligence cycle is characterized by manual collection and integration of data. Named Entity Recognition (NER) plays a fundamental role in Open Source Intelligence (OSINT) solutions when fighting crime. This paper describes the implementation of a NER-based focused web crawler under the EU FP7 Security Research Project CAPER (Collaborative information, Acquisition, Processing, Exploitation and Reporting for the prevention of organized crime). The crawler allows 1. to look for documents starting from a URL until a parametric depth of levels - also specifying a keyword that has to be contained in the page and in the related links - and 2. to look for a parametric number of documents starting from a keyword (entrusting the keyword search to one of the principal search engines, thus behaving as a meta-search engine). In addition, the crawler is able to retrieve only those documents that contain the information semantically relevant to the query (in other words: the required keyword with the required sense). This is achieved through the use of NER technologies. In this paper we present the CAPER NER-based Semantic Crawler, which has been proven to be a suitable tool for focused crawling, allowing LEAs to drastically reduce data collection and integration efforts.