[rabbitmq-discuss] RabbitMQ - Not able to get sets of messages in the same order

[Simon MacMullen]

Hi Srijanani.

On 10/06/10 15:53, Srijanani Srinivasan wrote:
> Thanks for your response. What you said makes sense. When the messages
> that have not been acknowledged are put back to the queue, then
> the other messages added in the meantime should be the ones to be
> de-queued first right in the FIFO manner? But like you said, i guess
> thats not how it works.

There isn't a guarantee about ordering for redelivered messages in the 
spec (since in more complex cases it's impossible to guarantee anyway), 
and Rabbit tries to do the most efficient thing rather than be FIFO in 
cases where it can.

> The same problem occurs, when I get a set of messages and don't
> acknowledge them and close the channel (in the event of any exceptions).
> When we get the messages, they do not come in the correct order.

Exactly.

> I guess, if i want to get sets of messages instead of one at a time, I
> must try to use the same channel. Is that right?

You could do that if it's convenient for you. Alternatively, you could 
prevent messages from being redelivered by making sure the server only 
sends you as many as you will actually process (if you know how many 
that is).

Oleg suggests you set the prefetch count to 1 - this will ensure that 
the server only sends you one message until you ack it (which will block 
if you're trying to process 100 at once).

So if you're currently reading in 100 messages and then acking all of 
them you could set the prefetch count to 100. The server will then allow 
you 100 unacked messages. You'd need to make sure you cancel before 
acking, otherwise the server can send you more messages between acking 
and closing the channel.

So to summarise:

channel.open
basic.qos(prefetch_count = 100)
basic.consume
(consume 100 messages)
basic.cancel
(ack 100 messages)

I'd be interested to know why you need to process messages in batches 
though.

Cheers, Simon

>
> On Thu, Jun 10, 2010 at 9:38 AM, Oleg Zhurakousky
> <ozhurakousky at vmware.com <mailto:ozhurakousky at vmware.com>> wrote:
>
>     Not sure about C# API, and how you are building the next set of
>     messages, but here is what I belive is happening.
>
>     Channel prefetch messages. That is different then invoking a
>     consumer. THis means that you might have 1000 prefetched messages in
>     your channel, while you are still processing 21st message. Whatever
>     you un-acknowledge is put back into the queue once the channel is
>     closed. AMQO does not define an ordering of messages for re-queueing.
>     One way to leverage this scenario is to set prefetchCount to 1. This
>     way there will never me more then one prefetched messages in your
>     channel.
>
>     Oleg
>
>     On Jun 10, 2010, at 10:24 AM, Srijanani Srinivasan wrote:
>
>
>     Hi,
>
>     I am using Rabbit MQ in C#. This is my scenario
>     1.               A separate process publishes messages to the queue
>     2.               Client has to read set of N messages from queue
>     3.               Process the N messages
>     4.               Acknowledge the N messages
>     5.               Repeat steps 2 to 4 continuously to process all
>     sets of messages
>     Under the same channel, I receive the messages and then process them
>     and then acknowledge them. The server process keeps publishing
>     messages. The problem I am facing is, when I try to get next set of
>     messages, they do not come in the same order as it was published by
>     the publishing process. The messages come in a random order. Only
>     the first set of messages comes in the correct order.
>
>     Does any one what is going wrong here? Is creating a new channel to
>     access the next set of messages not right? Or is there a problem
>     caused because of acknowledging multiple messages? Please help me
>     understand why this does not work correctly.
>
>     Below is the sample code:
>     while (true)
>                 {
>                     using (IModel getChannel = MQConnection.CreateModel())
>                     {
>                         // Create a consumer
>                         QueueingBasicConsumer consumer =
>     CreateQueueConsumer(getChannel, exchangeName, queueName);
>                         int numberOfMessages = 100;
>                         // Next Recieve
>                         List<object> msgSet =
>     GetNextSetOfMessages(consumer, getChannel, exchangeName, queueName,
>     numberOfMessages,             out finalDeliverytag);
>                         // Do some processing
>                         //Acknowledge finished messages by passing in
>     the delivery tag.
>                         // calls the method BasicAck with multiple
>     param=true
>                         if (finalDeliverytag > 0)
>                             AckFinishedMessages(exchangeName, queueName,
>     finalDeliverytag, getChannel);
>                         if (finalDeliverytag == 0)
>                             break;
>                     }
>                 }
>
>     Thanks for your help in advance!
>     Srijanani
>
>     <ATT00001..txt>
>
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss