[rabbitmq-discuss] dropped messages

Mon May 25 22:07:39 BST 2009

Mark,

smittycb10 wrote:
> however I still have the same problem 100,000's messages in and only
> 1,000's of messages out. Am I publishing and subscribing in the most
> efficent manner?

Can you send through the *complete* code? It's impossible to tell
otherwise why exactly you are getting the results you are seeing.

How are you measuring the rates? If on the sending side you are just
timing a loop of publishes then all you are measuring is the time it
takes to hand over the messages to the kernel. They can get buffered
there, in the network layer, in multiple layers at the broker, etc, etc.

Also, since AMQP's basic.publish is asynchronous, if the publishing 
connection is not terminated cleanly then some messages will inevitably 
get lost.

It's also worth remembering that RabbitMQ is a message *queuing* system, 
so one shouldn't be surprised if occasionally it queues some messages ;) 
What's more, because RabbitMQ's queuing is quite efficient, when you 
push the system hard, i.e. by feeding it messages as fast as you can, it 
will handle the incoming messages at a faster rate than it sends them to 
consumers.

For meaningful benchmark results it is therefore important to throttle
the producer, e.g. send messages at a fixed rate, monitor the rate at
which they are received by consumers, and slowly increase the producing
rate until the consumption rate starts falling behind.

>> Do I need to anything to broker?

Unlikely.

>> Is a message rate of 100,000 per second unreasonable?

For a single producer->broker->consumer chain that would be quite a high
rate - 10s of thousands of msgs/s is more realistic on stock hardware.

If you have several independent message streams then higher total
throughputs can be achieved when running on lots of cores.

> is there a max number of messages any one channel can have queued in
> it at one time?

No.

> I am currently trying to squeeze all the messages into one channel,
> should I be splitting work across channels and threads on the
> receiving side??

Having multiple channels and, even better, multiple connections, 
provides more opportunities for parellelism at the server. Whether or 
not that actually makes a positive difference depends on lots of factors.

Regards,

Matthias.