Complex Event Processing
Encyclopedia
Complex event processing (CEP) consists of processing many event
s happening across all the layers of an organization
, identifying the most meaningful events within the event cloud, analyzing their impact, and taking subsequent action in real time.
Complex event
processing refers to process states
, the changes of state exceeding a defined threshold of level, time, or value increment or just of a count as the event. It requires the respective event monitoring, event reporting, event recording and event filtering. An event may be observed as a change of state with any physical or logical or otherwise discriminated condition of and in a technical or economical system, each state information with an attached time stamp defining the order of occurrence and a topology mark defining the location of occurrence.
From these events the monitoring system may infer a complex event: a wedding. CEP as a technique helps discover complex events by analyzing and correlating other events: the bells, the man and woman in wedding attire and the rice flying through the air.
CEP relies on a number of techniques, including:
Commercial applications of CEP include algorithmic stock-trading
, the detection of credit-card fraud
, business activity monitoring
, and security monitoring.
(OI) solutions to provide insight into business operations by running query analysis against live feeds and event data. OI solutions use real-time data to collect and correlate against historical data to provide insight and analysis of the current situation. Multiple sources of data can be combined from different organizational silos to provide a common operating picture that uses current information. Wherever real-time insight has the greatest value, OI solutions can be applied to deliver the information and need.
In network management
, systems management
, application management and service management
, people usually refer instead to event correlation
. As CEP engines, event correlation engines (event correlators) analyze a mass of events, pinpoint the most significant ones, and trigger actions. However, most of them do not produce new inferred events. Instead, they relate high-level events with low-level events.
In artificial intelligence
, inferred information is typically produced by inference engine
s, e.g. rule-based reasoning engine
s. However, new information is usually not produced in the form of complex (i.e., inferred) events.
In the first situation, the car is moving and the pressure of one of the tires moves from 45 psi to 41 psi over 15 minutes. As the pressure in the tire is decreasing, a series of events containing the tire pressure is generated. In addition, a series of events containing the speed of the car is generated. The car's Event Processor may detect a situation whereby a loss of tire pressure over a relatively long period of time results in the creation of the "lossOfTirePressure" event. This new event may trigger a reaction process to note the pressure loss into the car's maintenance log, and alert the driver via the car's portal that the tire pressure has reduced.
In the second situation, the car is moving and the pressure of one of the tires drops from 45 psi to 20 psi in 5 seconds. A different situation is detected—perhaps because the loss of pressure occurred over a shorter period of time, or perhaps because the difference in values between each event were larger than a predefined limit. The different situation results in a new event "blowOutTire" being generated. This new event triggers a different reaction process to immediately alert the driver and to initiate onboard computer routines to assist the driver in bringing the car to a stop without losing control through skidding.
In addition, events that represent detected situations can also be combined with other events in order to detect more complex situations. For example, in the final situation the car was moving normally but suffers a blown tire which results in the car leaving the road and striking a tree and the driver is thrown from the car. A series of different situations are rapidly detected. The combination of "blowOutTire", "zeroSpeed" and "driverLeftSeat" within a very short space of time results in a new situation being detected: "occupantThrownAccident". Even though there is no direct measurement that can determine conclusively that the driver was thrown, or that there was an accident, the combination of events allows the situation to be detected and a new event to be created to signify the detected situation. This is the essence of a complex (or composite) event. It is complex because one cannot directly detect the situation; one has to infer or deduce that the situation has occurred from a combination of other events.
A computation-oriented CEP solution is focused on executing on-line algorithms as a response to event data entering the system. A simple example is to continuously calculate an average based in data on the inbound events.
Detection-oriented CEP is focused on detecting combinations of events called events patterns or situations. A simple example of detecting a situation is to look for a specific sequence of events.
However, the optimization of a business does not rely solely upon its individual, end-to-end processes. Seemingly disparate processes can affect each other significantly. Consider this scenario:
In the aerospace industry, it is good practice to monitor breakdowns of vehicles to look for trends (determine potential weaknesses in manufacturing processes, material etc). Another separate process monitors current operational vehicles life cycle and when appropriate decommissions at the end of their useful lives. Now one use for CEP is to link these separate processes, so that in the case of when the initial process (breakdown monitoring) discovers a malfunction based on metal fatigue (a significant event) an action can be created to exploit the second process (life cycle) to issue a recall on vehicles using the same batch of metal discovered as faulty in the initial process.
The integration of CEP and BPM must exist at two levels, both at the business awareness level (users must understand the potential holistic benefits of their individual processes) and also at the technological level (there needs to be a method by which CEP can interact with BPM implementation).
Computation-oriented CEP's role can arguably be seen to overlap with Business Rule technology.
Event
Event can refer to many things such as:* An observable occurrence, phenomenon or an extraordinary occurrenceA type of gathering:* A ceremony, for example, a marriage* A competition, for example, a sports competition* A convention...
s happening across all the layers of an organization
Organization
An organization is a social group which distributes tasks for a collective goal. The word itself is derived from the Greek word organon, itself derived from the better-known word ergon - as we know `organ` - and it means a compartment for a particular job.There are a variety of legal types of...
, identifying the most meaningful events within the event cloud, analyzing their impact, and taking subsequent action in real time.
Complex event
Event
Event can refer to many things such as:* An observable occurrence, phenomenon or an extraordinary occurrenceA type of gathering:* A ceremony, for example, a marriage* A competition, for example, a sports competition* A convention...
processing refers to process states
Process states
In a multitasking computer system, processes may occupy a variety of states. These distinct states may not actually be recognized as such by the operating system kernel, however they are a useful abstraction for the understanding of processes....
, the changes of state exceeding a defined threshold of level, time, or value increment or just of a count as the event. It requires the respective event monitoring, event reporting, event recording and event filtering. An event may be observed as a change of state with any physical or logical or otherwise discriminated condition of and in a technical or economical system, each state information with an attached time stamp defining the order of occurrence and a topology mark defining the location of occurrence.
Conceptual description
Among thousands of incoming events, a monitoring system may for instance receive the following three from the same source:- church bells ringing.
- the appearance of a man in a tuxedo with a woman in a flowing white gown.
- rice flying through the air.
From these events the monitoring system may infer a complex event: a wedding. CEP as a technique helps discover complex events by analyzing and correlating other events: the bells, the man and woman in wedding attire and the rice flying through the air.
CEP relies on a number of techniques, including:
- event-pattern detection
- event abstractionAbstractionAbstraction is a process by which higher concepts are derived from the usage and classification of literal concepts, first principles, or other methods....
- modeling event hierarchies
- detecting relationships (such as causality, membership or timing) between events
- abstracting event-driven processes
Commercial applications of CEP include algorithmic stock-trading
Algorithmic trading
In electronic financial markets, algorithmic trading or automated trading, also known as algo trading, black-box trading or robo trading, is the use of electronic platforms for entering trading orders with an algorithm deciding on aspects of the order such as the timing, price, or quantity of the...
, the detection of credit-card fraud
Credit card fraud
Credit card fraud is a wide-ranging term for theft and fraud committed using a credit card or any similar payment mechanism as a fraudulent source of funds in a transaction. The purpose may be to obtain goods without paying, or to obtain unauthorized funds from an account. Credit card fraud is also...
, business activity monitoring
Business activity monitoring
Business activity monitoring is software that aids in monitoring of business activities, as those activities are implemented in computer systems....
, and security monitoring.
Related concepts
One of many fields where CEP is used is Operational IntelligenceOperational Intelligence
Operational intelligence is a form of real-time dynamic, business analytics that delivers visibility and insight into business operations. Operational intelligence solutions run query analysis against live feeds and event data to deliver real-time, actionable information...
(OI) solutions to provide insight into business operations by running query analysis against live feeds and event data. OI solutions use real-time data to collect and correlate against historical data to provide insight and analysis of the current situation. Multiple sources of data can be combined from different organizational silos to provide a common operating picture that uses current information. Wherever real-time insight has the greatest value, OI solutions can be applied to deliver the information and need.
In network management
Network management
Network management refers to the activities, methods, procedures, and tools that pertain to the operation, administration, maintenance, and provisioning of networked systems....
, systems management
Systems management
Systems management refers to enterprise-wide administration of distributed systems including computer systems. Systems management is strongly influenced by network management initiatives in telecommunications....
, application management and service management
Service management
Service management is integrated into supply chain management as the joint between the actual sales and the customer. The aim of high performance service management is to optimize the service-intensive supply chains, which are usually more complex than the typical finished-goods supply chain...
, people usually refer instead to event correlation
Event Correlation
Event correlation is a technique for making sense of a large number of events and pinpointing the few events that are really important in that mass of information.-History:...
. As CEP engines, event correlation engines (event correlators) analyze a mass of events, pinpoint the most significant ones, and trigger actions. However, most of them do not produce new inferred events. Instead, they relate high-level events with low-level events.
In artificial intelligence
Artificial intelligence
Artificial intelligence is the intelligence of machines and the branch of computer science that aims to create it. AI textbooks define the field as "the study and design of intelligent agents" where an intelligent agent is a system that perceives its environment and takes actions that maximize its...
, inferred information is typically produced by inference engine
Inference engine
In computer science, and specifically the branches of knowledge engineering and artificial intelligence, an inference engine is a computer program that tries to derive answers from a knowledge base. It is the "brain" that expert systems use to reason about the information in the knowledge base for...
s, e.g. rule-based reasoning engine
Rule-based system
In computer science, rule-based systems are used as a way to store and manipulate knowledge to interpret information in a useful way. They are often used in artificial intelligence applications and research.-Applications:...
s. However, new information is usually not produced in the form of complex (i.e., inferred) events.
Example
A more systemic example of CEP involves a car, some sensors and various events and reactions. Imagine that a car has several sensors—one that measures tire pressure, one that measures speed, and one that detects if someone sits on a seat or leaves a seat.In the first situation, the car is moving and the pressure of one of the tires moves from 45 psi to 41 psi over 15 minutes. As the pressure in the tire is decreasing, a series of events containing the tire pressure is generated. In addition, a series of events containing the speed of the car is generated. The car's Event Processor may detect a situation whereby a loss of tire pressure over a relatively long period of time results in the creation of the "lossOfTirePressure" event. This new event may trigger a reaction process to note the pressure loss into the car's maintenance log, and alert the driver via the car's portal that the tire pressure has reduced.
In the second situation, the car is moving and the pressure of one of the tires drops from 45 psi to 20 psi in 5 seconds. A different situation is detected—perhaps because the loss of pressure occurred over a shorter period of time, or perhaps because the difference in values between each event were larger than a predefined limit. The different situation results in a new event "blowOutTire" being generated. This new event triggers a different reaction process to immediately alert the driver and to initiate onboard computer routines to assist the driver in bringing the car to a stop without losing control through skidding.
In addition, events that represent detected situations can also be combined with other events in order to detect more complex situations. For example, in the final situation the car was moving normally but suffers a blown tire which results in the car leaving the road and striking a tree and the driver is thrown from the car. A series of different situations are rapidly detected. The combination of "blowOutTire", "zeroSpeed" and "driverLeftSeat" within a very short space of time results in a new situation being detected: "occupantThrownAccident". Even though there is no direct measurement that can determine conclusively that the driver was thrown, or that there was an accident, the combination of events allows the situation to be detected and a new event to be created to signify the detected situation. This is the essence of a complex (or composite) event. It is complex because one cannot directly detect the situation; one has to infer or deduce that the situation has occurred from a combination of other events.
Types
Most CEP solutions and concepts can be classified into two main categories:- Computation-oriented CEP
- Detection-oriented CEP
A computation-oriented CEP solution is focused on executing on-line algorithms as a response to event data entering the system. A simple example is to continuously calculate an average based in data on the inbound events.
Detection-oriented CEP is focused on detecting combinations of events called events patterns or situations. A simple example of detecting a situation is to look for a specific sequence of events.
Integrating CEP with Business Process Management
Of course, rarely does the application of a new technology exist in isolation. A natural fit for CEP has been with BPM. BPM very much focuses on end-to-end business processes, in order to continuously optimize and align for its operational environment.However, the optimization of a business does not rely solely upon its individual, end-to-end processes. Seemingly disparate processes can affect each other significantly. Consider this scenario:
In the aerospace industry, it is good practice to monitor breakdowns of vehicles to look for trends (determine potential weaknesses in manufacturing processes, material etc). Another separate process monitors current operational vehicles life cycle and when appropriate decommissions at the end of their useful lives. Now one use for CEP is to link these separate processes, so that in the case of when the initial process (breakdown monitoring) discovers a malfunction based on metal fatigue (a significant event) an action can be created to exploit the second process (life cycle) to issue a recall on vehicles using the same batch of metal discovered as faulty in the initial process.
The integration of CEP and BPM must exist at two levels, both at the business awareness level (users must understand the potential holistic benefits of their individual processes) and also at the technological level (there needs to be a method by which CEP can interact with BPM implementation).
Computation-oriented CEP's role can arguably be seen to overlap with Business Rule technology.
Academic research
- Aurora (Brandeis University, Brown University and MIT)
- Borealis (Brandeis University, Brown University and MIT)
- Cayuga (Cornell University)
- ETALIS (Forschungszentrum Informatik Karlsruhe and Stony Brook University)
- Global Sensor Networks (EPFL)
- NiagaraST (Portland State University)
- PIPES (University of Marburg)
- SASE (UC Berkeley/UMass Amherst)
- STREAM (Stanford University)
- Telegraph (UC Berkeley)
- epZilla (University of Moratuwa)
- ACAIA.org Acoustic Event Detection (AED) (Acoustic Computing for AI/AmI Applications)
See also
- Event correlationEvent CorrelationEvent correlation is a technique for making sense of a large number of events and pinpointing the few events that are really important in that mass of information.-History:...
- Event-driven architecture — (EDA) is a software architecture pattern promoting the production, detection, consumption of, and reaction to events.
- Event Processing Technical SocietyEvent Processing Technical SocietyEvent Processing Technical Society is an inclusive group of organizations and individuals aiming to increase awareness of event processing, foster topics for future standardization, and establish event processing as a separate academic discipline....
— (EPTS) is an event processing community of interest - Event stream processingEvent Stream ProcessingEvent stream processing, or ESP, is a set of technologies designed to assist the construction of event-driven information systems. ESP technologies include event visualization, event databases, event-driven middleware, and event processing languages, or complex event processing...
— (ESP) is a related technology that focuses on processing streams of related data. - Real-time computingReal-time computingIn computer science, real-time computing , or reactive computing, is the study of hardware and software systems that are subject to a "real-time constraint"— e.g. operational deadlines from event to system response. Real-time programs must guarantee response within strict time constraints...
— CEP systems are typically real-time systems - Real time enterpriseReal Time EnterpriseReal-time enterprise is a concept in business systems design focused on ensuring organisational responsiveness that was popularised in the first decade of the 21st century. It is also referred to as on-demand enterprise...
- openPDCOpenPDCThe openPDC is a complete set of applications for processing streaming time-series data in real-time. The name stands for "open source phasor data concentrator" and was originally designed for the concentration and management of real-time streaming synchrophasors...
— A complete set of applications for processing streaming time-series data in real-time. - Operational intelligenceOperational IntelligenceOperational intelligence is a form of real-time dynamic, business analytics that delivers visibility and insight into business operations. Operational intelligence solutions run query analysis against live feeds and event data to deliver real-time, actionable information...
— Both CEP and ESP are technologies that underpin operational intelligence. - Pattern matchingPattern matchingIn computer science, pattern matching is the act of checking some sequence of tokens for the presence of the constituents of some pattern. In contrast to pattern recognition, the match usually has to be exact. The patterns generally have the form of either sequences or tree structures...
- RulecoreRulecoreRulecore is an ongoing research project investigating the use of different technologies to create software that support active behaviour. Especially active detection of event patterns...
- Streambase
- Sybase ESPSybaseSybase, an SAP company, is an enterprise software and services company offering software to manage, analyze, and mobilize information, using relational databases, analytics and data warehousing solutions and mobile applications development platforms....
- WebSphere Business EventsWebsphere Business EventsWebSphere Business Events is IBM's implementation of an event-processing engine.Event processing involves altering the existing server software in an organization to emit events whenever a notable event occurs. Event-processing software can then monitor these events and look out for certain...
- Esper - A Java and .NET complex event processing engine
- Oracle Complex Event Processing - A Rule-based Event Processing Engine
External links
- Separating the Wheat from the Chaff Article about CEP as applied to RFID, appeared in RFID Journal
- The Event Processing Technical Society
- Complex event processing: still on the launch pad in Computerworld
- Processing Flows of Information: From Data Stream to Complex Event Processing - Survey article on Data Stream and Complex Event Processing Systems
- S4 Distributed Event Processing Framework from Yahoo