Extending the SQL array concept to support scientific analytics

Abstract

Arrays are among those data types which contribute the most to Big Data -- examples include satellite images and weather simulation output in the Earth sciences, confocal microscopy and CAT scans in the Life sciences, as well as telescope and cosmological observations in Space science, to name but a few. Traditionally, the database community has neglected this, with the effect that ad-hoc implementations prevail. With the advent of NewSQL in recent years, however, the database scope has broadened, and array modelling and query support is seriously considered. Different models have been suggested, some of which are implemented or under implementation, and a consolidation of concepts can be observed. Consequently, integration of array queries into SQL is being addressed. We fill this gap by proposing a generic model, ASQL, for modelling and querying multi-dimensional arrays in ISO SQL. The model integrates concepts from the three major array models seen today: rasdaman, SciQL, and SciDB. It is declarative, optimizable, minimal, yet powerful enough for application domains in science, engineering, and beyond. ASQL has been implemented and is currently being discussed in ISO for extending standard SQL.