RTA: A Framework for the Integration of Local and Relational Open Data

Abstract

There are currently massive amounts of public data, also refereed to as open data, for example stock price data or weather data. However, such data is distributed in a variety of ways, such as downloadable files like CSV or XML files, or through API calls to web services. Each data source thus requires a specific workflow, making it a burden for the users to process and use this data. This barrier to use diminishes the openness of this data We thus propose the Remote Table Access (RTA) system, a simple and safe architecture for publishing, i.e. giving open read only access to relational data, and easily integrating it with the user's local data. RTA enables the user to query relational open data and their own local data seamlessly through a single SQL query. To allow this, we designed a three parties architecture featuring a client-side application, an optional server-side module and a "Public Table Library" (PTL). The client side application processes the RTA query and fetches the necessary data, the server side system acts as an agent between the remote database and the client, offering added security as well as scalability in terms of connections, and the PTL list all the published data and stores its access information. We implemented an early prototype of this architecture as a proof of concept. We validated it against two datasets, including data from the TPC-C benchmark and make it available1. Our results show the feasability of RTA and possible significant reduction of query processing time mainly because of the reduction on transmission volume by condition pushing and semijoin.