Indexing and Search of Correlated Business Events

Abstract

Complex event processing (CEP) is an emerging technology gaining a lot of momentum in research as well as in commercial products. CEP is a technique that utilizes event-driven IT-systems to monitor, optimize and steer todays business situations in real-time. Lots of contributions have been published on how to handle large amounts of event streams and how to extract meaningful knowledge out of often fine-grained events. From our experience we learned that complex use cases implemented with CEP need appropriate tool support to validate, understand and analyze the event processing done by the CEP software. We introduce the EventCloud system which is a generic event search and analysis tool integrating with CEP solutions in almost every domain. EventCloud is an end user application that allows complex search and analysis of correlated events. As CEP requires a highly specialized system architecture, the characteristics of events also have impacts on the infrastructure for indexing, searching and analyzing events. In this paper we present two approaches describing how to properly represent correlated events in persistent storages to allow performant event indexing and searching. Further we describe the "document-oriented" approach, where correlated events are managed in a full-text index and discuss the problems and pitfalls with its implementation in Apache Lucene. Finally we compare different full-text engines, namely Apache Lucene and Microsoft SQL Server 2005, to analyze their performance and features for event indexing and search