GNOSIS- query-driven multimodal event processing for unstructured data streams

Abstract

This paper presents GNOSIS, an event processing engine to detect complex event patterns over multimodal data streams. GNOSIS follows a query-driven approach where users can write complex event queries using Multimodal Event Processing Language (MEPL). The system models incoming multimodal data into an evolving Multimodal Event Knowledge Graph (MEKG) using an ensemble of deep neural network (DNN) and machine learning (ML) models and applies a neuro-symbolic approach for event matching. GNOSIS follows a serverless paradigm where its different components act as independent microservices and can be deployed across different nodes with optimized edge support. The paper demonstrates two multimodal use case queries from Occupational Health and Safety and Accessibility domain.