Missing Data Recovery for Downstream Learning in Low-Cost Environmental Sensor Networks

Abstract

The development of low-cost sensor networks (LCSNs) has facilitated the generation of vast amounts of environmental monitoring data in air quality monitoring stations. State-of-the-art research has focused on developing robust models to detect and mitigate events which may impact societies. Making adequate predictions is important for meeting the sustainability needs of future generations. Therefore, quality data is required to produce accurate results for downstream analytics and visualisations. When dealing with large amounts of data, it is however common to observe numerous amounts of missing data. This is a challenge for data miners because various methods for data analysis only work well on complete datasets. A traditional approach to handling missing data is to discard instances of missing values and only use complete cases for analysis. However, research has shown that this approach is not practical especially when large amounts of data are missing. This led to an increased need to develop strategies for replacing missing values with plausible values through imputation. This study presents an imputation strategy called k−BFMVI for recovering missing values before training downstream regression models for low cost sensor calibration. Experiments simulated missingness from 10% to 40% using MCAR and MAR mechanisms and the performance of the proposed technique was measured against state-of-the-art techniques. Overall, the proposed algorithm recorded the best imputation accuracy as opposed to existing techniques and showed significant improvements on downstream learning.