Main Content

High-Performance Event Processing on Modern Hardware

Technological advancements have changed the way data is received, stored and queried. Traditional database concepts have been challenged by new application scenarios such as the Internet of Things (IoT), e-Science and industry 4.0. Data originating from these applications is usually produced continuously, inherently temporal in nature and associated queries require fast processing to avert dangerous situations. Therefore, a crucial novel aspect for handling these applications is the ability to consume potentially infinite streams of data, evaluate continuous queries and deliver answers in (near) real time.

One of the key technologies for dealing with these requirements is Complex Event Processing (CEP). CEP systems are designed to answer complex queries on streamed temporal event data through powerful operators such as pattern matching. Applied to the scenarios above, CEP queries can be used to specify patterns describing intricate events such as alarms while the system delivers low latency responses on ever changing data streams. This enables the design of quick reaction pipelines for critical components.

With the growing maturity of the applications above, more complex use cases suitable for CEP are emerging. In addition to fast queries on the latest data, there is an increasing demand for combining historical and current data in a reciprocal way - e.g., for adaptive queries, for testing new query parameters and to analyze situations lasting longer periods of time. Meeting the processing demands for these use cases while maintaining the ability for continuous quick reactions on data streams is a major challenge. However, novel co-processors such as GPUs offer opportunities to increase throughput for a variety of streaming applications. Furthermore, advancements in modern storage technologies such as SSDs showcase a lot of potential to not only store, but efficiently index a large amount of streamed data. 

In order to enable the efficient realization of mature CEP applications, we propose a system that deals with both low latency requirements of Complex Event Processing and high-throughput analysis of historical (event) data. Our core research objective is to combine both approaches through the following measures:

  • The introduction of the situation calculus into the world of CEP, which allows for native long-term event analysis.
  • The development of new querying strategies for our event database system (ChronicleDB) in order to use historical data for real time processing purposes.
  • The integration of modern hardware (GPUs, SSDs) into a CEP system for increased processing efficiency of event queries.
  • The exploration of novel CEP technologies through prototype development in a reactive infrastructure monitoring scenario at the European Organization for Nuclear Research (CERN). 

As part of a large research initiative on Scalable Data Management for Future Hardware (DFG Priority Program 2037), we are also exploring related research opportunities beyond the goals listed above.

Staff

Additional Information

  • Publications

    Michael Körber, Nikolaus Glombiewski, Andreas Morgen, Bernhard Seeger:
    TPStream: low-latency and high-throughput temporal pattern matching on event streams.
    Distributed and Parallel Databases (2019): 1-52.

    Christian Beilschmidt, Johannes Drönner, Nikolaus Glombiewski, Christian Heigele, Jana Holznigenkemper, Anna Isenberg, Michael Körber, Michael Mattig, Andreas Morgen, Bernhard Seeger:
    Pretty Fly for a VAT GUI: Visualizing Event Patterns for Flight Data.
    DEBS 2019: 224-227

    Michael Körber, Jakob Eckstein, Nikolaus Glombiewski, Bernhard Seeger:
    Event Stream Processing on Heterogeneous System Architecture.
    DaMoN 2019: 3:1-3:10

    Nikolaus Glombiewski, Bernhard Seeger, Goetz Graefe:
    Waves of Misery After Index Creation.
    BTW 2019: 77-96

    Michael Körber, Nikolaus Glombiewski, Bernhard Seeger:
    TPStream: Low-Latency Temporal Pattern Matching on Event Streams.
    EDBT 2018: 313-324

Further links