[rabbitmq-discuss] message stats from a given node

Matthias Radestock matthias at lshift.net
Tue Jan 19 07:57:01 GMT 2010


Brian,

Brian Sullivan wrote:
> I was wondering if there are any good ideas out there for tracking 
> message rates on a given cluster (in our use case, it's mostly 
> topic-routed messages).  We have been using RabbitMQ for a while now, 
> and have been limited to tracking stats via a consumer that effectively 
> subscribes to '#' and simply breaks down these stats by topic.

So the information you are interested in is "average number of messages 
per second published, per routing key and time period", correct?

> This works ok until the message volumes get pretty heavy - the node that 
> the client connects to has to be routed all messages on the cluster, and 
> this breaks down the scalability model for RabbitMQ - eventually that 
> node will be a bottleneck.

One thing you could do at the AMQP level is introduce stat-collecting 
proxies between the publishers and the brokers. Take a look at the 
RabbitMQ tracer in src/com/rabbitmq/tools/Tracer.java in the 
rabbitmq-java-client repo for an example of a proxy.

Another route is via the "pluggable exchange types" we have under 
development - see the bug22169 branch in the rabbitmq-server hg repo. 
That would allow you to define a custom exchange type that works just 
like an ordinary topic (or other) exchange but collects stats on 
publishes on a per-node basis. In the simplest case this could just dump 
a timestamp and routing key to a file, leaving the actual statistics 
calculations to some other program. Or, at the other extreme, the custom 
exchange could compute per-node stats itself and, furthermore, 
participate in the collation of stats from multiple nodes, using the 
distributed computing features Erlang puts at your disposal.


Regards,

Matthias.




More information about the rabbitmq-discuss mailing list