[rabbitmq-discuss] message stats from a given node

Tue Jan 19 07:57:01 GMT 2010

Brian,

Brian Sullivan wrote:
> I was wondering if there are any good ideas out there for tracking 
> message rates on a given cluster (in our use case, it's mostly 
> topic-routed messages).  We have been using RabbitMQ for a while now, 
> and have been limited to tracking stats via a consumer that effectively 
> subscribes to '#' and simply breaks down these stats by topic.

So the information you are interested in is "average number of messages 
per second published, per routing key and time period", correct?

> This works ok until the message volumes get pretty heavy - the node that 
> the client connects to has to be routed all messages on the cluster, and 
> this breaks down the scalability model for RabbitMQ - eventually that 
> node will be a bottleneck.

One thing you could do at the AMQP level is introduce stat-collecting 
proxies between the publishers and the brokers. Take a look at the 
RabbitMQ tracer in src/com/rabbitmq/tools/Tracer.java in the 
rabbitmq-java-client repo for an example of a proxy.

Another route is via the "pluggable exchange types" we have under 
development - see the bug22169 branch in the rabbitmq-server hg repo. 
That would allow you to define a custom exchange type that works just 
like an ordinary topic (or other) exchange but collects stats on 
publishes on a per-node basis. In the simplest case this could just dump 
a timestamp and routing key to a file, leaving the actual statistics 
calculations to some other program. Or, at the other extreme, the custom 
exchange could compute per-node stats itself and, furthermore, 
participate in the collation of stats from multiple nodes, using the 
distributed computing features Erlang puts at your disposal.

Regards,

Matthias.