Developing a data pipeline to improve accessibility and utilization of Charlottesville's Open Data Portal

Abstract

To improve democratic engagement between the people and the government, the city of Charlottesville put forward a proposition to construct an online portal that would contain data from the city departments that is considered public by nature. This move was intended to promote the ease of access to data pertinent to ongoing policy debates in the city and incentivize the public to contribute to the policy-making process with informed participation. Such efforts, while successful at their start, have gradually stagnated, and the end objective of the portal has not been reached. In this paper we identify possible reasons for this stagnation – inconsistent formatting of the datasets, variables that are not meant for human legibility, and limited data with disproportional representation from the city departments. We then propose a data pipeline that serves as a tool to extract utility from the data. It does so by converting the datasets into a consistent format, merges the datasets, and allows for creation of simple visualizations. The pipeline acts as a link between the raw data published by the government units and the city by increasing its interpretability and legibility and outputting results that are easily relatable to the policy issues at hand. We demonstrate this by analyzing datasets for crime and real estate and relating our findings to the affordable housing debate.