Data preparation: Art or science?

Abstract

Data preparation is often cited as the most time consuming phase of a Knowledge Discovery and Data Mining (KDDM) process. This is attributed to the fact that this phase is highly dependent on the expertise of the analyst. Although process models exist for KDDM the description of their phases of the process focus on outlining what must be done but often do not detail how this should be done. While there is some research in addressing the how of the phases, the data preparation phase is thought to be the most challenging and is often described as an art rather than a science. The tasks defined in this phase are thought to be highly dependent on the expertise of the analyst and the context. While we are of the view that there will always be an art to data preparation we will demonstrate that the science can actually enhance the art. We further contend that as more research of this kind is published, that demonstrates a variety of data preparation techniques that enhance the data mining process, the more effective will be the science of data preparation.