[rabbitmq-discuss] Lightweight query for queues?

[Simon MacMullen]

Hi Ian.

On 03/05/11 19:18, Ian Ragsdale wrote:
> So the updated management plugin seems to have gotten rid of the
> crashes I was seeing, which is great - thanks to the rabbitmq devs
> for a quick response!

Cool.

> One useful thing about that thread was figuring out just how
> resource-intensive the management plugin is when querying the API for
> queues.  On my server with around 6000 queues, I can run 'rabbitmqctl
> list_queues' and get results in a second, but if I query the API for
> the list of queues it pegs one of the CPUs for upwards of 30 seconds,
> returning over 8MB of data.

That seems really wrong. While the management plugin hasn't received 
much performance tuning it should in theory be capable of being faster 
than rabbitmqctl in such scenarios, since the management plugin 
maintains its own statistics database while rabbitmqctl has to go off 
and talk to each queue individually.

And indeed, for a trivial test (declare 6k empty queues, time 
rabbitmqadmin -f raw_json list queues > /dev/null), I get 1.9 seconds on 
my workstation. That should be improved (preliminary profiling suggests 
it's spending a lot of time *sorting* the list!), but I'm not sure why 
you're seeing something so much slower.

I know this would have been very slow prior to fixing the bug you 
encountered (due to writing very large error messages in the logs) - 
sorry if this is patronising but you're definitely seeing the slow 
response with the fixed version?

Also, if you set force_fine_statistics to false for the 
rabbit_management_agent application - e.g with a 
/etc/rabbitmq/rabbitmq.config file like:

[{rabbit_management_agent, [{force_fine_statistics, false}]}].

  - does it make any difference?

<snip>

> So, what I'd like to be able to do is:
>
> 1. Allow specifying which data needs to be returned, so that more
> expensive statistics can be avoided if necessary. 2. Allow filtering
> the list of queues to return data for with a search term

<snip>

> Is this something you guys would consider?

Probably not (1) right now, since none of these statistics are supposed 
to be even slightly expensive, they should all just be hash lookups. (2) 
is somewhat interesting, probably simpler to implement and more 
generally useful. But I'd like to find out why you're seeing this run so 
slowly first.

Cheers, Simon

-- 
Simon MacMullen
Staff Engineer, RabbitMQ
SpringSource, a division of VMware