[rabbitmq-discuss] Messages not showing up

[Gavin M. Roy]

On Monday, March 14, 2011 at 11:28 AM, Vukasin Toroman wrote: 
> Hi everyone,
> 
> we have been using rabbitmq for a while now to route documents through a processing workflow. each step is represented by one or more processes working on one queue. the messages are small (maybe a few k) but there are quite a few of them (multiple thousand per second). This has been working fine for a while but then we have noticed that the rates (each step of the processing workflow delivers stats) do not add up. namely we seem to be losing messages (around 200 messages/sec) and this is quite discerning. The reason is that these messages do not show up in the queue and are also not visible as being recieved and un-acked. we are using non-durable queues and a combination of java and python processing clients.
How are you quantifying the counts? Do you keep track on your own outside of RabbitMQ message counts on your consumers and publishers? 
> oh yes, this only happens after the system has been running for a while and a traffic spike has occurred. the rabbitmq process also seems to be gaining memory. 
> 

I have found that if I am over saturating a single node with regard to message throughput, I see a pattern where memory use increases without being able to reconcile where it is used. During spikes like this, I did see abnormal behavior with messages as well. 

In my case, I was seeing this behavior when throughput exceeded ~25k messages per second being published into a single node on a single exchange while consuming from multiple queues bound to the exchange connected to other nodes in the rabbitmq cluster. 

To address this, I changed the publishing strategy and added a few nodes to the cluster. I still use the same exchange and routing key structure, however I am now publishing into the exchange across multiple nodes instead of one. I have not seen any issues since this change and am now operating at 2x the previous message velocity across the cluster.

Hope this helps,

Gavin


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110314/1f89ff54/attachment.htm>