Topic Set Size Design and Power Analysis in Practice

Abstract

Topic set size design methods provide principles and procedures for test collection builders to decide on the number of topics to create. These methods can then help us keep improving the test collection design based on accumulated data. Simple Excel tools are available for such purposes. Post-hoc power analysis tools, available as simple R scripts, can help IR researchers examine the achieved power of a reported experiment and determine future sample sizes for ensuring high power. Thus, for example, underpowered user experiments can be detected, and a larger sample size can be proposed. If used appropriately, these Excel and R tools should be able to provide the IR community with better experimentation practices. The main objective of this tutorial is to let IR researchers familiarise themselves with these tools and understand the basic ideas behind them.