An open source cyberinfrastructure for collecting, processing, storing and accessing high temporal resolution residential water use data

Abstract

Collecting and managing high temporal resolution (< 1 minute) residential water use data is challenging due to cost and technical requirements associated with the volume and velocity of data collected. It is well known that this type of data has potential to expand our knowledge of residential water use, inform future water use predictions, and improve water conservation strategies. However, most studies collecting this type of data have been focused on the practical application of the data (e.g., developing and applying end use disaggregation algorithms) with much less focus on how the data were collected, retrieved, quality controlled, and managed to enable data visualization and analysis. We developed an open-source, modular, generalized cyberinfrastructure system to automate the process from data collection to analysis. The system has three main architectural components: first, the sensors and dataloggers for water use monitoring; second, the data communication, parsing and archival tools; and third, the analyses, visualization and presentations of data produced for different audiences. For the first component, we present a low-cost datalogging device, designed for installation on top of existing, analog, magnetically driven, positive displacement, residential water meters that can collect data at a user configurable time resolution interval. The second component consists of a system developed using existing open-source software technologies that manages the data collected, including services and databasing. The final element includes software tools for retrieving the data that can be integrated with advanced data analytics tools. The system was used in a single family residential water use data collection case study to test the scalability and performance of its functionalities within our design constraints. Testing with a base system configuration, our results show that the system requires approximately six minutes to process a single day of data collected at a four second temporal resolution for 500 properties. Thus, the system proved to be effective beyond the typical number of participants observed in similar studies of residential water use and would scale well beyond this even with the modest system resources we used for testing. All elements of the cyberinfrastructure developed are freely available in open source repositories for re-use.