An approach to monitoring and managing clustered servers as a single entity

January 23, 2007

Often we need to be able to monitor and manage clustered servers as a single entity. This is an outline of an approach I’ve used before to monitor and manage a clusterable enterprise server product which had very demanding RASP and throughput requirements. It is based on JMX, but allows for an aggregated view of the mbeans (monitoring attributes and management operations) in the cluster i.e. it allows for things to be examined in a meaningful way at the cluster (or service) level, while at the same time being able to view an individual nodes value for that attribute.

It also enables an operation to be executed once for the whole cluster e.g. flush a cache, rather than having to manually do that for each server instance in turn. This can be achieved using a standard single server aware mbean console (a limitation of most off the shelf open source ones).

The kind of aggregation appropriate for an individual attribute can be easily configured, and a view can be presented at the cluster level and at the individual servers (node) level. Different attributes require different kinds of aggregation. For example, the aggregation of the number of items in a given cache across the cluster would be the sum of all the individual cache sizes, but at the same time we also want to be able to see the value for an individual node. In the case of message throughput the aggregation might be an average of all the individual servers throughputs, and we might also want a count across the cluster of messages processed to date, as well as those statistics on an individual node basis. For another attribute it might not be sensible to aggregate it, so we might just want the individual values for each node to be shown together. The point is that a sensible “cluster” view of a given attribute differs.

The basic idea is to instrument the classes that house the stuff you want to expose as normal using an MBean interface. So each node in the cluster exposes all its stuff as an individual server. Additional metadata is added to the MBean interface using variables (static final Strings), which describe what is to be done with the individual attributes and operations. I say static final Strings because this annotation is Java 1.4.x compatible, but if you’re using Java 5 then of course you can use “real” annotations.

The mbeans are registered with the mbean server via a façade – lets call it MBeanServerFacade. The facades job is to interrogate the mbeans and extract metadata using introspection. A simple convention is followed for the variable names in order to associate them with the monitoring attributes and management operations. The values of the strings then describe how the attribute is aggregated (and other useful things). The metadata could be described somewhere else, but the advantage of putting it on the mbean interface is that its easy to develope, and it lives with the attributes, so when the mbean changes the metadata can be easily changed with it. Also, it is really simple to add these annotated mbeans so there is no barrier to adding instrumentation to the server (and for clustered servers setups you can never know enough about what is going on under the covers). MBeanServerFacade writes the metadata to a simple database table, if that particular type of mbean hasn’t already been registered. It also writes the host address of each node in the cluster to another table.

The next element in the architecture is a broker process. This should (but does not have to) live separately to the cluster, so on a standalone server instance. The MBeanServerBroker reads (and periodically reloads) the descriptions in the tables populated by MBeanServerFacade. So the broker has a full description of all the relevant mbeans available and all the servers where they can be found. MBeanServerBroker is used as an intermediary between the various MBeanServers and the calling application. Using the metadata MBeanServerBroker creates JMX dynamic mbeans which expose an appropriate aggregate view of the individual mbeans across the cluster. Because it uses dynamic mbeans, the aggregate mbeans can also expose the individual attribute values (and which server they belong to). When an aggregate operation is invoked from the console, MBeanServerBroker runs that operation on each node in the cluster. The other important job of the broker is caching the returned mbean information. This is because JMX requests are expensive both in terms of the caller (waiting/blocking) and the component exposing the mbeans (doing the work of gathering the mbean info etc.). The console remains responsive because its hitting a cache, and the broker updates the cached values frequently. It’s the dynamic mbeans exposed by MBeanServerBroker that the console looks at.

Here is an example of an annotated mbean interface:

interface MyStatsMBean {

/** attributes */

int getCacheSize();
int getAvgJmsReadsPerSecond();
int getMaxInsertMessageTime();
int getMinInsertMessageTime();
int getUniqueValue();

/** annotations */

String getCacheSize_AM = AggregateMethod.SUM;
String getAvgJmsReadsPerSecond_AM = AggregateMethod.AVG;
String getMaxInsertMessageTime_AM = AggregateMethod.MAX;
String getMinInsertMessageTime_AM = AggregateMethod.MIN;
String getUniqueValue_AM = AggregateMethod.LIST;

In this example we want to see the sum of cache sizes across the cluster, the average JMS reads per second across all nodes, the slowest message insert into the database, as well as the quickest, and a list of individual values for an attribute that it does not make sense to aggregate.


2 Responses to “An approach to monitoring and managing clustered servers as a single entity”

  1. rockwell Says:

    I feel far more men and women require to read this, quite excellent info.

  2. I just book marked your blog site on Digg and StumbleUpon. I enjoy
    reading your commentaries.
    I also run a blog site that might be of interest that I’d love to share
    with you. I recently stated that?-
    When it comes to dealing with difficult eBay buyers, a single problem that commonly arises is with your current auction listings. It is not uncommon to get an eBay message from a buyer requesting that you lower your price on one of your listed items. In fact, it is not uncommon for an eBay buyer to mention that other eBay sellers have the same items listed for less. If and when this ever happens to you personally, you will wish to keep your cool. You should simply state that that you are unable to lower your asking price. Kindly suggest to the eBay buyer in question that if they can be looking for more affordable prices, they should consider doing business with an eBay seller who offers more affordable merchandise. The decision as to whether or not you need to lower your eBay prices is yours to make, but you should never feel pressured into doing so.
    Feel no cost to poke around my weblog and leave a comment. Us
    quality bloggers have to stick together! /

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: