Establishing a framework for the ethical and legal use of web scrapers by cybercrime and cybersecurity researchers: learnings from a systematic review of Australian research

Abstract The Internet has become an increasingly attractive location for collecting data about cyber threats, driven by the abundance of quality data available and accessible online. As such, researchers and practitioners have turned to automated data collection technologies (ADCT), including ‘web crawlers’ and ‘web scrapers’, to study these threats. The rapid proliferation of ADCT has meant directions for their ethical and legal operation have been slow to adapt, with no clear guidelines regulating their use for research. This article identifies the relevant ethical and legal frameworks guiding the deployment of ADCT in Australia for cybersecurity research. This is accomplished through a systematic review of research within this context, coupled with ethical and jurisprudential analysis. We argue that the use of ADCT can be both ethical and legal, but only where mitigating measures are implemented. We provide a series of practical directions to guide researchers and practitioners when navigating this novel terrain.