History 1: http://complexevents.com/wp-content/uploads/2008/02/1-a-short-history-of-cep-part-1.pdf History 2: http://complexevents.com/wp-content/uploads/2008/07/2-final-a-short-history-of-cep-part-2.pdf Terms: http://www.complexevents.com/wp-content/uploads/2011/08/EPTS_Event_Processing_Glossary_v2.pdf
Event Stream Intelligence with Esper and NEsper
Esper is a component for CEP and ESP applications, available for Java as Esper, and for .NET as NEsper.
Esper and NEsper enable rapid development of applications that process large volumes of incoming messages or events. Esper and NEsper filter and analyze events in various ways, and respond to conditions of interest in real-time.
Technology Introduction
Complex Event Processing, or CEP, is technology to process events and discover complex patterns among multiple streams of event data. ESP stands for Event Stream Processing and deals with the task of processing multiple streams of event data with the goal of identifying the meaningful events within those streams, and deriving meaningful information from them. Real-time OLAP (online analytical processing) and continuous query are also terms used frequently for this technology.
The Esper engine has been developed to address the requirements of applications that analyze and react to events. Some typical examples of applications are:
* Business process management and automation (process monitoring, BAM, reporting exceptions, operational intelligence)
* Finance (algorithmic trading, fraud detection, risk management)
* Network and application monitoring (intrusion detection, SLA monitoring)
* Sensor network applications (RFID reading, scheduling and control of fabrication lines, air traffic)
One year ago I penned Event Processing in Twitter Space, and today parts of the net are buzzing about Twitter.
In a nutshell, Twitter is a one-to-many communications service that uses short messages (140 chars or less). Following on the heels of the blogging phenomena, Twitter has been primarily used for microblogging and group communications.
Twitter, and Twitter-like technologies, has great promise in many areas. For example, you could be subscribed to the @tsunamiwarning channel on your dream island vacation and get instant updates on potential disasters. A team of people working in network management could subscribe to the @myserverstatus channel and receive updates on their health of their company IT services. Passengers could subscribe to the @ourgatestatus channel and follow up-to-date information on their fight.
Twitter was created to answer the simple question, “What are you doing now?”
Bruce makes an interesting comment on business rules too: that “routing logic in process gateways” are not “business rules”. That doesn’t really make sense: for sure some gateways will be process-housekeeping decisions of little interest to the business user, but others will surely embed business-critical decisions. On the other hand, it has long been acknowledged that a best practice for BPM is to delegate such business decisions to a managed decision service - hence the explicit new business rule (aka decision) task in BPMN 2.0. And,in the CEP world, for tools like TIBCO BusinessEvents to invoke a decision managed by its Decision Manager tool.
The main characteristic to be aware of in these tools is that BE is primarily rule-based (using an embedded rule engine), whereas BW and iProcess are orchestration / flow engines. In BE we can use a state diagram to indicate a sequence of states which may define what process / rules apply, but this is really just another way of specifying a particular type of rules (i.e. state transition rules).
The main advantages to specifying behavior as declarative rules are:
Handling complex, event-driven behavior and choreography
Iterative development, rule-by-rule
The main advantages of flow diagrams and BPMN-type models are:
Ease of understanding (especially for simpler process routes)
Process paths are pre-determined and therefore deemed guaranteeable.
In combination these tools provide many of the IT capabilities required in an organization. For example, a business automation task uses BW to consolidate information from multiple existing sources, with human business processes for tasks such as process exceptions managed by iProcess. BE is used to consolidate (complex) events from systems to provide business information, or feed into or drive both BW and iProcess, and also monitors end-to-end system and case performance.
On Event Processing Agents implies a “new” event processing reference architecture with terms like,
(1) simple event processing agents for filtering and routing,
(2) mediated event processing agents for event enrichment, transformation, validation,
(3) complex event processing agents for pattern detection, and
(4) intelligent event processing agents for prediction, decisions.
Frankly, while I generally agree with the concepts, I think the terms in On Event Processing Agents tend to add to the confusion because these concepts in On Event Processing Agents are following, almost exactly, the same reference architecture (and terms) for MSDF, illustrated again below to aid the reader.
The success of Service-Oriented Architecture (SOA) has created the foundation for information
and service sharing across application and organizational boundaries. Through the use of SOA,
organizations are demanding solutions that provide vast scalability, increased reusability of
business services, and greater efficiency of computing resources. More importantly,
organizations need agile architectures that can adapt to rapidly changing business requirements
without the long development cycles that are typically associated with these efforts. Event-Driven
Architecture (EDA) has emerged to provide more sophisticated capabilities that address these
dynamic environments. EDA enables business agility by empowering software engineers with
complex processing techniques to develop substantial functionality in days or weeks rather than
months or years. As a result, EDA is positioned to enhance the business value of SOA.
The purpose of this white paper is to describe the approach employed to overcome the significant
technical challenges required to design a dynamic grid computing architecture for a US
government program. The program required optimization of the overall business process while
maximizing scalability to support dramatic increases in throughput. To realize this goal, an
architecture was developed to support the dynamic placement and removal of business services
across the enterprise.
JT has posted his view on rules and decisions and how they relate. Given that James talks more about services than events, I thought it would be worth reviewing his post from both a Complex Event Processing and a TIBCO BusinessEvents event processing platform perspective.
”Decision Services:
Support business processes by making the business decisions that allow a process to continue.
Support event processing systems by adding business decisions to event correlation decisions (they are often called Decision Agents in this context).
Allow crucial and high-maintenance parts of legacy enterprise applications to be externalized for reuse and agility.
Can be plugged into a variety of systems using Enterprise Service Bus approaches.”
Folks who have been in event processing fields like network management (NMS) or security management for many years have a very high expectation for processing complex events. Most of the network and security management platforms on the market have basic rule-based processing available “out of the box” and most of these platforms have had the capability to process events in near-real time for decades. Adding a new “rules-based event processing platform” to the network and security management software mix does little to add any additional capability and certainly does not solve any nagging complex detection problems.
- leave anything related to transport, communication to other layers- use this revised CEP to express and execute event-relevant logic, the purpose of which is to translate the ambient events into relevant business events- have these business events trigger business processes (however lightweight you want to make them)- have these business processes invoke decision services implemented through decision management to decide what they should be doing at every step- have the business processes invoke action services to execute the actions decided by the decision services- all the while generating business events or ambient events- etc.
A common task in many event processing systems is to detect patterns of events.
If combined, these patterns will eventually form a situation consisting of multiple patterns over time.
So basically a detected instance of a situation is a specific sequence of events.
CEP module receives or intercepts a flurry of events and processes them with the objective of figuring out what those events are relevant for; it triggers the appropriate business processes or decision services
BPM module receives the request for a given process to be applied to a higher level entity (an application, a document...); it automates the steps defined in the business process
BRMS module is invoked with a given context to apply business rules; it makes a business decision
Function answers the question --- what is being done?Technique answers the question -- how something being done?
Application answers the question --- what is the problem being solved?
ExamplesBusiness Activity Monitoring (BAM) is an application type, it solves the problem of controlling the business activities in order to optimize the business, deal with exceptions etc...Business Rules are type of technique --- which can be used to infer facts from other facts or rules (inference rules) , or to determine action when event occurs and condition is satisfied (ECA rules) and more (there are at least half a dozen types of rules, which are techniques to do something).Event Processing is really a set of functions which does what the name indicates -- process events --- processing can be filtering, transforming, enriching, routing, detect patterns, deriving and some more.
EDA is more a manifestation of finite state machines going all the way back to Alan Turing. Old_State + Event = Some_Action + New_State. It is the simplest, yet most powerful way to design robust systems. I only wish more people would give it due consideration.
A very old implementation example is I/O interrupts (hardware events) for asynchronous I/O - real time event handling which enabled multitasking operating systems.
Many want to use web services for everything now and at times it is hard to convince people that other messaging schemes and standards are a better fit for some problems.
Rule-processing is just a style of computation. Of course it is used in BRMS, but it is also used in CEP. CEP systems typically employ rules-based processing to infer higher-order events by matching patterns across many event streams within the event ‘cloud’. BRMS’s use rule processing to match patterns within data tuples representing business-orientated data. CEP systems may support the use of advanced analytics to manage predictive analysis, reasoning under uncertainty and other requirements in relation to the event cloud. Some of the better BRMS’s offer similar analytics in regard to processing business data.
L. Stojanovic, S. Sen, J. Ma, и D. Riemer. Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, стр. 385--386. New York, NY, USA, ACM, (2012)
D. Anicic, S. Rudolph, P. Fodor, и N. Stojanovic. Proceedings of the 5th international conference on Rule-based reasoning, programming, and applications, стр. 138--153. Berlin, Heidelberg, Springer-Verlag, (2011)
Y. Xu, N. Stojanovic, L. Stojanovic, и T. Schuchert. Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, стр. 379--380. New York, NY, USA, ACM, (2012)
Y. Verginadis, I. Patiniotakis, N. Papageorgiou, и R. Stühmer. The Semantic Web: ESWC 2011 Workshops, том 7117 из Lecture Notes in Computer Science, Springer Berlin / Heidelberg, 10.1007/978-3-642-25953-1_16.(2011)
M. Lauras, Y. Verginadis, R. Stühmer, и F. Benaben. Proceedings of the 6th IEEE Int. Conf. on Digital Ecosystems and Technologies for Complex Systems, Environment, and Service Engineering IEEE-DEST 2012, (2012)
A. Barthe, F. Benaben, S. Truptil, J. Lorre, и H. Pingaud. Workshop Advanced results in MDI/SOA innovation in IWEI 2011 Third International IFIP Working Conference Interoperability and Future Internet for Next-Generation Enterprises, Stockholm, (2011)
A. Barthe, M. Lauras, и F. Benaben. 8th International Conference on Information Systems for Crisis Response and Management : From early-warning systems to preparedness and training, Lisbon, (2011)
A. Barthe, F. Benaben, S. Truptil, J. Lorre, и H. Pingaud. Workshop Ädvanced results in MDI/SOA innovation" in IWEI 2011 Third International IFIP Working Conference "Interoperability and Future Internet for Next-Generation Enterprises", Stockholm, (2011)