The liabilities of robots.txt

Abstract

This paper explores the legal implications of violating robots.txt, a technical standard widely used by webmasters to communicate restrictions on automated access to website content. Although historically regarded as a voluntary guideline, the rise of generative AI and large-scale web scraping has amplified the consequences of disregarding robots.txt directives. While previous legal discourse has largely focused on criminal or copyright-based remedies, we argue that civil doctrines, particularly in contract and tort law, offer a more balanced and sustainable framework for regulating web robot behavior in common law jurisdictions. Under certain conditions, robots.txt can give rise to a unilateral contract or serve as a form of notice sufficient to establish tortious liability, including trespass to chattels and negligence. Ultimately, we argue that clarifying liability for robots.txt violations is essential to addressing the growing fragmentation of the internet. By restoring balance and accountability in the digital ecosystem, our proposed framework helps preserve the internet’s open and cooperative foundations. Through this lens, robots.txt can remain an equitable and effective tool for digital governance in the age of AI.