YANCEES: Yet ANother Configurable Extensible Event Service

Introduction

In this project, we study the use of configurable and extensible software architectures to provide versatility to event notification services (a.k.a. publish/subscribe infrastructures). It fills the gap found in current notification servers that provide few or no support for extending, customizing and adding new functionality to their core set of features. As a response to that gap, we developed YANCEES, an extensible and configurable framework that provides a versatility layer on top of current publish/subscribe infrastructures, allowing their extension, programming and configuration to attend the demand of specific application domains. Examples of such extensions include subscriptions with event correlation, different notification policies, protocols for mobile applications integration, ad-hoc network protocols; event persistence services, security and others. This is accomplished by the combined use of extensible languages and plug-in oriented architectures, along main publish/subscribe design dimensions defined by event, subscription, notification, protocol and resource models. In other words, versatility is achieved by:

1. An open, extensible and configurable architecture where plug-ins can be installed
2. Extensible event, notification, subscription and protocol languages

Internally, YANCEES is a framework that combines:

1. Dynamic parsers that allocate plug-ins to cope with the provided subscription and notification commands/policies.
2. Input and output filters
3. Special pluggable services
4. Configuration managers and builders
5. Adapters that can support existing publish/subscribe networks
6. A core that integrates those featues.

Motivation

Many distributed applications interact by the exchange of events. For the point of view of the applications, notification services provide a distributed implementation of the implicit invocation pattern, intermediating the communication between publishers of information and their consumers (subscribers). In such systems, at one side, publishers produce pieces of information that are sent (or published) to a logically-centralized service (a.k.a.notification service); whereas information consumers (or subscribers) express their interest on those pieces of informationby means of subscriptions. Subscriptions are usually represented as logical expressions on the content, type and/or order of events. More advanced systems allow more expressive constructs, permitting the correlation of events, for example. This approach allows the insulation between producers and consumers of information, which copes with the dynamism, heterogeneity, scalability and changeability requirements of many distributed applications. The publish/subscribe communication style also facilitates the combination and integration of information from different sources as they get produced.

For having such characteristics, event notification servers have been used as the basic communication infrastructure in a large set of application domains. These domains include user and software monitoring, testing, CSCW, awareness, workflow management systems, mobile applications, enterprise integration, software engineering environments and so on. Due to their increasing popularity and the peculiarities of each application domain, different functinality has been demanded from such infrastructures. 

Even though a big set of commercial and research notification services are currently available, they are not usually designed for evolution and configurabilty. In other words, they provide poor programability, configurability and dynamic change support. From a software engineer perspective, before YANCEES, developers had basically three options when using notification services in their applications:

• Use what we call “one-size-fits-all” or monolythic approaches, such as adopted by CORBA Notification Service (CORBA-NS), READY or JMS. These infrastructures strive to address new applications requirements by providing, in advance, a very comprehensive set of features. Besides their inability to scale (down) to more simple applications, they are not designed with extensibility in mind. As applications evolve demand new requirements, the infrastructure itself cannot evolve to address these new requirements.

• Use specialized notification services, designed for different application domains. Also motivated by the poor versatility of current standardized solutions such as CORBA-NS or JMS, many notification services have been developed to provide novel but specific functionalities demanded by different applications. Examples of such specialized systems include KHRONIKA and CASSIUS which are specially designed to support groupware and awareness applications; or YEAST and GEM which are specialized in advanced event processing for local networks applications, or JEDI, which first studies strategies to support mobility in publish/subscribe systems. Moreover, those infrastructures are usually implement incompatible and non-programmable solutions, which in a big organization result in incompatibilty and interoperability problems, since  different notification services may need to co-exist and inter-operate.
  

• In a third and emerging category, notification services such as Siena or Elvin can be used. They provide a minimal core of features in a content-based network. A compromise between expressiveness and performance is achieved in favor of its scalability, providing an optimized content-based event routing network. Due to their emphases on fast and scalable routing of events, they provide a simple but efficient filtering language. Configurability programability is not their primary concerns, being the responsibility of the application to implement extra services such as event persistence, and advanced event correlation, for example.

Configurabilty: Specialization versus Generalization. Therefore, in the development of event driven distributed applications, developers face the dilemma of specialization versus generalization: to use a generalized infrastructure, that can support and integrate different applications, but that may not provide all the necessary functionality (in terms of subscription language, notification policies and other points of variance in notification services); or to use specialized event-based infrastructures for each application domain, having “the right tool for the right problem”, but loosing the uniformity and integration of a common solution. For example, in the development of awareness tools, one can use either a comprehensive solution such as CORBA Notification Service (CORBA-NS) that provides a large set of features and services, or a domain specific solution such as CASSIUS, that provides services for awareness applications. That’s important to note that infrastructures as CORBA-NS, even though are very comprehensive in its set of features, do not provide specific support for all the requirements of many specific application domains. For example, in the case of awareness applications, CORBA-NS does not provide event source hierarchy representation, browsing and discovery. CASSIUS, on the other hand, provides these features and allows the easy discovery of information sources and the subscription for events from different source components. If CORBA-NS were used in an awareness application demanding this feature, this service would need to be implemented by the system developers, in the application layer.

Processing distribution. Another problem of the currently available event-based infrastructures is the poor support for selection and customization of where to execute the  event processing computation. Few notification services, such as READY, provide this support. READY extends the CORBA-NS, allowing subscription processing to be done in the client side.

Infrastructure Evolution. Moreover, current event-based infrastructures lack mechanisms that allow the programming (or evolution) of their functionality. The only extension mechanism is usually the (understanding and) change of the notification service source code, or the implementation of the missing functionality as part of the application itself. Additionally, due to restrictions in their event or subscription models, the addition of new functionality may become a very time consuming and difficult task.

Scalabilty. Different applications demand different functionality sets from the publish/subscribe infrastructure. Moreover, they have different hardware requirements, requiring the support for device-constrained configurations such as PDAs, or more demanding server-side applications.  Hence, infrastructures that allow the scale (up and down) of their features to different application requirements are not available.

Interoperabilty. Interoperability issues may also be a problem. In large organizations, for example, if multiple notification services are used in the support for different application domains, their integration may be an issue. They are usually incompatible with one another and do not easily interoperate, as a result of different event syntax and semantic, as well as subscription formats.

Dynamism. In order to be useful, the configurability and programmabilty aspects need to  be provided in a way that does not inerrupt the publish/subscribe infrastructure. This requires the ability to dinamically configure/upgrade the system as  required by the addplicatoins.

Usability. Another important aspect to be considered is usabilty. A notification service must be easy to use, in the point of view of the programmers, providing services that allow, for example, the registration and discovery of new event souces, as a way to enhance the subscription ability of the service.

Functionality evolution in publish/subscribe systems

In order to design a generic and programmable framework, the application domain must be understood. This requirements analysis must lay out the main activities involved in the process being automated, allowing the generalization of the solution through the implementation of an object-oriented framework. In other words, a framework must provide extensibilty along the main dimension of the problem being generalized.

In the YANCEES framework, the programability points are inspired in the framework proposed by (Rosenblum and Wolf 1997), which captures the most relevant dimensions (or models) in the design of publish/subscribe services. In this framework, the object model describes the components that receive notifications (subscribers) and generate events (publishers). The event model describes the representation and characteristics of the events; the notification model is concerned with the way the events are delivered to the subscribers; the observation model describes the mechanisms used to express interest in occurrences of events; the timing model is concerned with the casual and temporal relations between the events; the resource model defines where, in the distributed system architecture, the observation and notification computations are located, as well as how they are allocated and accounted; finally, the naming model is concerned with the location of objects, events, and subscriptions in the system.

For the implementation of YANCEES, we consider a variation of this model proposed by (Cugola, Nitto, and Fuggetta 2001), with the addition of a new model, the protocol. The protocol model is defined as a way to address the need to capture other forms of interaction with the notification service that goes beyond the common publish/subscribe interaction. This is necessary to incorporate the ability to handle different functionality other than the common publish and subscribe activities, and that require the interaction of the publish/subscribe core with other components of the system. Examples of protocols that may be necessary are: guaranteed delivery, mobility and roaming protocols, security, event source discovery and other possible interaction mechanisms with the service.

The YANCEES strategy to provide programability and configurability is to implement these models as programmable points along an object-oriented framework. An extension in this model is represented by programmable languages and protocols, whose implementation is provided by user-defined plug-ins, services and filters.

Overview

Currently, YANCEES is bult on top of a generic and replaceable pub/sub core. It provides an programability layer, that allows the use of different subscription, notification and event models with the development of plug-ins and language extensions (using XML). 

As highlighted in the picture above, plug-ins are used to extend the event, notification, subscription and protocol models. These models are implemented as sets of languages defined in XML. Examples of plug-ins include event correlation expressions (sequence detection, rules, temporal and causality constructs), push and pull notification delivery policies, move-in/move-out protocol primitives to support mobile clients, event ontology and persistence services, and so on. An alternative picture of the overall approach is presented as follows.

YANCEES Event-Notification Service Architecture

In a high-level perspective, YANCEES provides a framework that combines plug-ins (in red) with language extensions (in orange), along the main event processing steps. Along with those dimensions, the protocol dimension is also provided, supporting the implementation of different interaction mechanisms with the server.

Under the hood, YANCEES provides a set of parsers that combine the extensions in the languages (which express the design models), with their semantic implementation, the plug-ins. Internally, the framework provides auxiliary extension points (filters, services and adapters) that are used to support the implementation of more complex policies and services. Adapters are used to provide interoperability and support for different publish/subscribe substrates (currently Siena and Elvin). As shown in the picture below, each parser (in orange) is responsible for extending different design dimensions of a publish/subscribe system (notification, protocol and subscription).

The server-side architecture of the YANCEES framework is presented as follows.

The components of the system are divided in main categories: parsers (orange), auxiliary parser components (service manager and plug-in manager) and a component configuration manager (all in green), and the instances allocated by this set of components (plug-ins, filters and services in red).

Parsers (in orange) are used to dynamically allocate plug-ins in response to messages sent to YANCEES API, mainly publish and subscribe commands. They use plug-in instances (in red), that are dynamically allocated by the Plug-in manager (green). Plug-ins are hierarchically composed, forming evaluation trees that extend the basic functionality of the event dispatcher core. There is one parser to each main design dimension. Plug-in instances (in red) extend the core subscription and notification languages provided by the dispatcher adapter and the dispatcher component (in blue). Protocol plug-ins interact with the service instances and filters in response to protocol messages. The architecture configuration manager deals with  the dynamic installation of components and their distribution between client an server side. The whole components can be installed in the client-side, moving event processing to the client, or can be shared in the server-side, an intermediate approach balancing the location of the plug-ins can also be achieved.

Filters and services are used to enhance the functionality provided by the plug-ins. Filters process events in the input and output streams of the framework, allowing the implementation of security policies, event type checking and so on; services can be used to provide access points to shared resources such as databases, data models, and so on.

The following figure shows the flow of messages in the system and the dynamic allocation of plug-ins in response to the messages provided (other components are omitted here for simplicity).

The arrival of a message (subscription arrow in the picture) triggers the appropriate parser (Subscriber API), which identifies the XML tags of the message. In the example above, we assume a subscription message  having two parts, a subscription expression and a notification policy. The subscriber API splits the message in their two components, forwarding each part to the appropriate parser, in this case, the notification and the subscription parsers. Each parser allocates plug-in instances to handle the commands expressed in each part of the message. An evaluation tree, composed of plug-in instances, is then created, in memory, to handle (or interpret) that particular message. The tree is composed of notification and subscription nodes. The leaves of this tree are special plug-ins that subscribe to events in the dispatcher. Plug-is more close to the top of the tree perform higher-level computations, such as event sequence detection. Once the tree is parsed, events coming from the dispatcher that matches the leaves subscriptions are processed, in a bottom-up approach, through this structure, having the results sent to the subscriber as notifications. Before delivering the notifications, the message passes through output filters that enforce global policies in the system. Note that while subscriptions are handled on demand, and are associated to unique subscribers, filters are applied to all the events of the system.

The basic parsing algorithm, which is also applied to the notification, subscription and protocol models, is illustrated as follows (example with subscription language and parser):

Plug-ins associated to each tag of the message (subscription in our example) are allocated according to their disposition in the XML model (or document) where they are represented. Each plug-in instance handles one or more XML tags in the message being parsed. Events are sent upward in the subscription tree as the plug-ins expressions are evaluated.

Versatility Summary

The following table summarizes the event notification design dimensions supported by YANCEES and describes how to extend each one of them in our approach.

Configurability Summary

The configurabilty is provided by YANCEES configuration language, which allows the selection of specific plug-ins, filters and services to be used by each YANCEES instance. The configuration file is provided when the service is started. Pluig-ins interdependencies are checked and enforced by the configuration manager at bootstrap time.

Dynamism Summary

Implementation Status

The YANCEES prototype was implemented and tested with Java (J2SDK1.4.2 and WSDP1.2 - which provides XMLSchema validation support for the DOM parser). Currently, the framework supports the core mechanisms presented in the previous sections: input and output filters, subscription, notification and protocol plug-ins, and shared services, which can be configured at boot time with the use of configuration description files. The system was ported to work using Siena and Elvin. It also supports our own fast topic-based dispatcher. The communication between client stubs and the server is implemented using Java RMI protocol.

The following plug-ins are implemented: event sequence detection, pull and push notification policies, polling protocol, a simple persistence service (in memory), a fast multiplexer dispatcher (topic-based), and many other features.

The service is multithreaded and supports many simultaneous subscriptions and publications as non-blocking operations. Input and output buffers were implemented to improve overall throughput and performance.

In the next releases of YANCEES, we plan to improve the architecture with:

• Improved end-user subscription language and interface (as opposed to bare XML subscription language only)
• Finer-grained distribution of components between client and server
• Address issues related to integration of different event notification networks, for example, Elvin, Siena and CORBA-NS - integration layer. In other words, improve the interoperability and generality of the adapters.
  
• Validate the system with other applications. Currently, we are using it to support a security awareness project and a software monitoring tool.
• Run YANCEES on a Pocket PC. Current experiments with YANCEES on PocketPC are described in the document: 
  Configuring Java (J9) on a Pocket PC with RMI, SWT and SVG using CDC Personal Profile: some experiences and issues
  
  

References

Download

More current source code is available in the CVS repository.

Examples, installation instructions, user manual and configurations are under construction at the YANCEES manual page (under construction).

This software is distributed according to the following LICENSE.TXT.
This software was tested with J2SDK1.4.2 and Java API for XML processing available at SUN WSDP1.2 (web services pack) or higher.

YANCEES Bundles

For your convenience, we provide some examples of YANCEES bundled in different configurations. Those buldes include only a Yancees-SNAPSHOT.jar file with the publisher and  subscriber source codes. windows and unix shell scripts are provided as well as the configuration file used to customize YANCEES.

Minimal YANCEES: (August 2006 snapshot) Configured with a fast switch core, the subscription language has only one command, <require>, which filters messages with the specified attribute name.

Siena YANCEES: (August 2006 snapshot) Configured with Siena as the routing core, it provides siena configuration language with an extra sequence detector plug-in. This is the version used in our example above.

YANCEES prototype version 0.7

June 8th 2004 - The third stable public release of the YANCEES prototype is available. Previous releases can be downloaded here. The Java implementation of the prototype can be downloaded as follows:

Release notes:

     This version supports:

Plans for the next version:

In a medium term, we are also planning to develop:

Roberto Silveira Silva Filho
Irvine, CA. March 18th, 2004.

Publications and Technical Reports

Acknowledgements

This research has been sponsored by NSF (grants 0133749, 0205724, 0326105, 0527729,0524033), an IBM Eclipse technology exchange grant, and the Intel Corporation.