[rabbitmq-discuss] Stress testing RabbitMQ

Sat May 14 12:32:41 BST 2011

On Sat, May 14, 2011 at 01:07:49PM +0200, Alvaro Videla wrote:
> I tried creating an alternate_fanout exchange, storing the bindings in
> a gb_tree, dicts, and plain lists… still I can see much improvement
> compared to the default fanout exchange. My goal was to avoid the
> mnesia:dirty_select() done at rabbit_router:match_routing_key.

Have you tried benchmarking mnesia:dirty_select? I think you'll find
it's actually pretty quick.

Also, in the context of a fanout exchange with many thousands of
bindings, you're basically going to be reading the entire binding table
anyway, so it make no difference whether you're able to use indices or
not.

Also rabbit_route is an orderedset table. The key of #route is #binding.
#binding is (obviously) just a tuple, and we fill as much as possible
from left to right in the tuple as the matchhead:

-record(binding, {source, key, destination, args = []}).

match_routing_key(SrcName, [RoutingKey]) ->
    MatchHead = #route{binding = #binding{source      = SrcName,
                                          destination = '$1',
                                          key         = RoutingKey,
                                          _           = '_'}},

So we will have {binding, SrcName, RoutingKey, '$1', '_'} as the key to
query on an orderedset table. That's not inefficient.

mnesia:dirty_read is very fast indeed. Please benchmark it. I think
you'll find it's basically the same speed as ets:match_object which is
actually what it'll be doing under the bonnet.

I have not looked into your benchmark, however, what size msgs are you
using? I suspect that if the binary fragments end up under 64 bytes then
that is the cause of memory explosion: each msg is being _copied_ rather
than reference counted to each queue. If you try with msgs with a bigger
payload, I wonder whether you see the same behaviour.

Matthew