[rabbitmq-discuss] multiple message listeners on a single reply queue

Thu Feb 6 18:07:12 GMT 2014

Hi

We are beginners in rabbitmq. We use rabbitmq 3.2.2. We use spring amqp.

I recently observed an issue with multiple listeners on the same reply
queue when performance testing

our application.

Here is the test scenario:

50 Concurrent users post a request to Service F to create an entity A. The
message contains a user id and a timestamp. Each concurrent user posts
roughly two messages per second.

F publishes this message to a non-durable queue via a topic exchange (using
convertSendAndReceive) and waits for the response on a fixed reply queue
ReplyQ1.

Service A picks up this message and creates this entity (basically an entry
in a table) after

validating that the user id wihin that message is valid.

For the validation, Service A sends this message to another non-durable
queue via a topic exchange (convertSendAndReceive) and waits for a response
on

another fixed reply queue, ReplyQ2.

A validating service V picks up this user id, validates it (basically a
select from a db table) and then sends the response back to ReplyQ2.

Service A picks this up, based on the message, creates the entity and then
sends response to ReplyQ1.

All messages are non-persistent. No concurrency or prefetch settings
configured on the listeners.

*CreateEntity A Request -> Service F -> Q1 -> Service A -> Q2 - Service V*

*Service V-> ReplyQ2 -> Service A (creates entity A if user valid) ->
ReplyQ1 -> Service F*

F, A and V all live within the same tomcat instance. When a second tomcat
instance was started with service A and V, I started to notice warning
messages in tomcat logs:

*WARN : org.springframework.amqp.rabbit.core.RabbitTemplate - Reply
received after timeout for b3fd4f57-0403-4704-8f16-8c3055ca0cd9*

It looked like messages were not picked up by the correct listener from the
validation response queues. Rabbitmq logs did not show any errors. From the
management console i saw 40-60 messages always staying in ready state  in
Q1, with a few in unacked state.

Response times were very high for this create Entity A request (10 seconds)
and only about 5 transactions were processed per second. Tests were run in
AWS on m1.large instances.

High response time was not caused by network latency. Network latency was
really small.

On testing with separate reply queues for the user id validation response,

(ReplyQ21 and ReplyQ22 instead of ReplyQ2), response times improved (1
second) and more transactions (roughly 70) were processed per second.

Is this a good design approach for this scenario? Or is there a better way
to handle this ?

Please let me know if you need any other information.

Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140206/f8d9b68d/attachment.html>