[rabbitmq-discuss] problem seen with multiple listeners on a single reply queue

[TaraGopi N]

Hi
We are beginners in rabbitmq. We use rabbitmq 3.2.2. I recently observed an
issue with multiple listeners on the same reply queue when performance
testing
our application written in Java.

Here is the test scenario:
50 Concurrent users post a request to Service F to create an entity A. The
message contains a user id and a timestamp. Each concurrent user posts
roughly two messages per second.

F publishes this message to a non-durable queue via a topic exchange (using
convertSendAndReceive) and waits for the response on a fixed reply queue
ReplyQ1.

Service A picks up this message and creates this entity (basically an entry
in a table) after
validating that the user id wihin that message is valid.

For the validation, Service A sends this message to another non-durable
queue via a topic exchange (convertSendAndReceive) and waits for a response
on
another fixed reply queue, ReplyQ2.

A validating service V picks up this user id, validates it (basically a
select from a db table) and then sends the response back to ReplyQ2.
Service A picks this up, based on the message, creates the entity and then
sends response to ReplyQ1.

CreateEntity A Request -> Service F -> Q1 -> Service A -> Q2 - Service V
Service V-> ReplyQ2 -> Service A (creates entity A if user valid) ->
ReplyQ1 -> Service F

F, A and V all live within the same tomcat instance. When a second tomcat
instance was started with service A and V, I started to notice warning
messages in tomcat logs:

WARN : org.springframework.amqp.rabbit.core.RabbitTemplate - Reply received
after timeout for b3fd4f57-0403-4704-8f16-8c3055ca0cd9

It looked like messages were not picked up by the correct listener from the
validation response queues.

Response times were high for this create Entity A request (10 seconds) and
only about 5 transactions were processed per second. Tests were run in AWS
on m1.large instances.
High response time was not caused by network latency. network latency was
really small

On testing with separate reply queues for the user id validation response,
(ReplyQ21 and ReplyQ22 instead of ReplyQ2), response times improved (1
second) and more transactions (roughly 70) were processed per second.

Is this a good design approach for this scenario? Or is there a better way
to handle this ?

All messages are non-persistent. No concurrency or prefetch settings
configured on the listeners. Please let me know if you need any other
information.

Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140205/42110461/attachment.html>