A Hierarchical Database of One Million Websites

Abstract

As part of a broader cyber-policy experiment on the Use and Abuse of Personal Information, we are seeking efficient methods to generate a hierarchical and malleable database of over one million websites for use in a future large-scale, semi-automated establishment of fake online accounts. Available directories of reputable Internet sites are often incomplete, outdated, or not well categorized. This paper describes the design and challenges associated with a custom web scraper to refine Curlie [1], an online repository of websites, into a concise, readable format. The scraper recursively and distributively crawls Curlie for unique URLs and plain-text names and parses them into our database. We will use the hierarchy functionality of this new database to answer future research questions focused on website stewardship of personal information (PI). This data normalization challenge is one of many we have encountered in the larger open-source intelligence (OSINT) Use and Abuse (U&A) collection framework.