A methodology for managing database and code changes in a regression testing framework

Abstract

Large system development typically involves changes to code and database components. In these environments, testing continues to be the biggest challenge. Since release cycles include both database and code changes, thorough testing of existing schemas and code base is paramount. Code needs to be tested with a static database, database changes need to be tested on a static code base, and finally, both changes need to be tested together. In this paper we present the testing framework built during the development of Morgan Stanley's enterprise-wide time-series database, Horizon. The database, built using Kx's KDB+/Q, holds both tick-by-tick data originating from real-time feeds as well as periodic data received at regular intervals from vendors. The plant consists of several hundred cooperating processes which connect to real-time feeds, create in-memory databases, persist in-memory databases to disk on a regular basis, manage data set locations, and service client connections. At present, the database plant's size is approximately 1PB with an anticipated growth of .5PB/year.