Exploring an AI Data Dredging Birthday Paradox

Abstract

In the era of AI and Data Science, the extensive use of big databases for various purposes, including crime investigations, medical studies, and general population profiling, leads increasingly to the possibility of random database matches driven merely by coincidence akin to the famous birthday paradox. As databases swell in size and complexity, we show in this paper that under some circumstances the likelihood of coincidental matches between seemingly unrelated entries increases dramatically. These extraneous matches can inadvertently mislead investigators and analysts, ultimately resulting in incorrect source attributions.Applying the mathematics of generalized birthday problems, this paper uses an expository approach to delve into the intricacies of data dredging across diverse data sets, emphasizing the need for caution when interpreting results obtained through post-hoc analysis. We explore the potential consequences of relying on post-facto data-driven storytelling, highlighting the dangers of attributing meaning to even matches that occur with seemingly extraordinary odds.