Spam and Botnet Reputation Randomized Control Trials and Policy

Designing randomized control trials (RCT) of reputational effects of spam and botnet rankings as proxies for Internet security has interesting challenges. These challenges are related to the policy issues such reputation is intended to address. Building on preliminary results and the public SpamRankings.net top 10 rankings per country by spam volume from two anti-spam blocklists (see TPRC 2012 [1] and 2011 [2] papers), formal RCT experiments provide another level of evidence. However, using RCT with thousands of organizations in treatment and control groups raises numerous difficulties in non-homogeneous legal and organizational regimes and methods of active disclosure of comparable rankins across peer groups. Fortunately most of these difficulties can be turned to advantages, and all have policy implications. These complications compared to RCTs of more traditional econometric one-shot surveys with single publication arise because the subject of these field experiments is the live Internet in real time with ongoing updated treatments. The experimental treatments themselves act as information security (infosec), since their purpose is to use reputation to cause internal improvements in infosec in treated companies. treatments thus must adapt to changes in conditions in the Internet as they happen. Like other infosec, to be effective the treatments must also be portable across departments within treated organizations plus customers and investors, and the experimental team itself crosses Economics, Information Systems, and Computer Science. If the experiments demonstrate statistical evidence that this reputational approach works, such results will provide a new policy approach of reputational rankings, plus the beginnings of tools to apply that approach, ranging from the public treatments themselves to drilldowns into underlying details of the symptoms causing good or bad reputation. Difficulties encountered include: 1) Differing sensitivities of different blocklists to spam from certain sources; sensitivities that change over time as the blocklists adapt to new miscreant behavior. Approach: A weighted composite ranking based on both spam volume and spamming address count from at least two different blocklists. 2) Heterogeneity of legal regimes and other characteristics across countries. Approach: Initial experiments within a single country (the U.S.), perhaps followed by clustered RCT using countries as clusters. 3) Availability of organizational characterization information for stratification by industry (finance, medical, etc.) and within industry (ISPs or hosting, telephone company or cable company, etc.). Approach: Start with the U.S., for which this information is relatively readily available in homogeneous form. 4) Public visibility is necessary for reputation so that customers and investors of treated organizations can see the treatments, yet limits flexibility of experimental treatments, since an ongoing, regularly updated treatment once deployed is hard to retract. Approach: Start with a subset of the universe of spamming organizations and deploy more treatments for other organizations later, plus potential additional treatments for already-treated organizations, while tuning existing treatments like product releases. 5) Spammers or bot herders could choose to migrate away from treated organizations to untreated (control) organizations, interfering with independence of treated and control groups. Approach: Use botnet volume and address data to observe whether this actually happens (potential future work). 6) Miscreants may actively retaliate with DDoS or other attacks. Approach: Harden the treatment websites by hosting them in a cloud provided by a very large organization. 7) Many of the most relevant and we think potentially effective features of this work are nonobvious to many persons skilled in various arts indigenous to at least seven major markets the work must reach, in academia, inside the treated organizations, and in governance. Designing marketing materials and interaction methods to make the nonobvious obvious is a major part of this work. Specifically, drawing connections from spam as a proxy for underlying security issues to organizational benefits of reputational rankings to societal benefits of active disclosure is quite a challenge for a tiny research organization simulating the sales and marketing (and engineering) departments of a large corporation. Approach: Model on rankings comprehensible to everyone (sports scores), use analogies, emphasize benefits, tailor to specific markets where necessary, provide writeups on the most nonobvious features, such as active vs. passive/disclosure. This series of experiments is supported by NSF grants 1228990 and 0831338, and the usual disclaimers apply.