An Introduction to Dynamic Data Quality Challenges

Abstract

We live in an evolving world. As time passes, data changes in content and structure, and thus becomes dynamic. Data quality, therefore, also becomes dynamic because it is an aggregate characteristic of data itself. Thus, our evolving world and Internet of Things (IoT) presents renewed challenges in data quality. IoT data is teeming with multivendor and multiprovider applications, devices, microservices, and automated processes built on social media, public and private datasets, digitized records, sensor logs, web logs, and much more. From intelligent traffic systems to smart healthcare devices, modern enterprises are inundated with a daily deluge of dynamic big data. The primary characteristics of big data are volume, velocity, and variety [Abadi et al. 2014]. Techniques for managing volume and velocity have been under development for decades. While some work has been done on variety, integrating and analyzing data from diverse sources and formats still presents challenges. For example, much of the big data deluge is structured and much of it is not. This single dimension of variety inherent in today’s IoT clearly illustrates there is no “silver bullet” and one size does not fit all [Abadi et al. 2014; Stonebraker and Cetintemel 2005, 2015]. It is important to note there are many other dimensions of variety beyond structure. We must consider possibilities arising from analyzing data in a dizzying range of data types found in varying time frames of differing granularity from diverse sources in our evolving and streaming world. Structure is but one example illustrative of many more general challenges that we use in this article to introduce dynamic data quality.