Ensuring real-time data integrity in smart building applications: A systematic end-to-end comprehensive pipeline evaluated in numerous real-life cases

Abstract

In this study, we propose a comprehensive, end-to-end data healing pipeline, developed and tested in real buildings facing diverse severity problems. This pipeline is designed to ensure the accuracy and reliability of smart building data in a fast and responsive manner, using computationally lightweight statistical approaches for outlier detection and LightGBM for data imputation. Both methods are optimized for low computational cost, making them ideal for real-world scenarios requiring immediate feedback and capable of handling very large datasets efficiently. Our system is designed to operate automatically, capable of applying real-time data processing and periodic model updates without manual intervention. It was evaluated using Key Performance Indicators over nine weeks across five smart buildings in the EU, revealing discrepancies in performance across different time periods and buildings. These findings highlight the need for tailored data healing strategies for varying dataset sizes, ultimately enhancing data quality for more reliable analyses and informed decision-making. The implementation of this pipeline contributes to more accurate energy usage, improved occupant comfort, and more efficient building operations, supporting the broader goals of sustainability and energy efficiency in smart buildings.