A Technology Recommender System Based on Web Crawling and Natural Language Processing

Abstract

The goal of this study is to develop a prototype for a recommendation system that could assist individuals and organizations without in-depth knowledge of technology products and services to choose an appropriate tool/service that best suits their needs. Recommendation systems, as they are popularly known, are personalized information filtering systems which are usually integrated into various consumer and commercial applications. These personalized systems play a vital role, especially when the user is unsure of what to search for. Our work focuses in particular on web-based recommendation systems for end-users in academia and industries who need recommendations for their software tools and services. This paper focuses on extracting information from the web using Apache Nutch, an open-source web crawler which extracts data from websites widely used for software recommendations. The information extracted is indexed in Elasticsearch, whose Natural Language Processing (NLP) and text mining capabilities are applied to provide appropriate recommendations to the end-users. Kibana dashboards and visualizations are used to visualize the recommendations in a format that is conducive for the end-users.