<div>Thanks for your response Simon.</div><div><br></div><div>I was not using Basic.QOS method to set the prefetch count so far. After your suggestion, I modified the code to follow the same order like you said and I tested it. I see that the order is preserved, when a new channel/consumer is created for accessing the next set of messages. But, if we don't acknowledge a certain set of messages, like you said, the order is not preserved as they will be getting re-delivered.</div>
<div><br></div><div>In my application, we have high volume of messages that are going to be coming in through the queue. And we will have to do some calculation and store data in database. We have realized earlier that saving the data one by one is time consuming, so we will take one set of say 5000 messages and then do the calculations and store it the database in one database call. So basically, we do not want to process one by one to save database calls. Another reason, is that there could be a lot of duplicate messages, so we will pick 5000 and do the work only for the unique ones.</div>
<div><br></div><div>Is there any performance gain if we ACK a set of messages instead of ACKing one by one, as we are reducing the number of responses sent from the client side when we do bulk ACks?</div><div><br></div><div class="gmail_quote">
On Tue, Jun 15, 2010 at 7:45 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com">simon@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi Srijanani.<div class="im"><br>
<br>
On 10/06/10 15:53, Srijanani Srinivasan wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks for your response. What you said makes sense. When the messages<br>
that have not been acknowledged are put back to the queue, then<br>
the other messages added in the meantime should be the ones to be<br>
de-queued first right in the FIFO manner? But like you said, i guess<br>
thats not how it works.<br>
</blockquote>
<br></div>
There isn't a guarantee about ordering for redelivered messages in the spec (since in more complex cases it's impossible to guarantee anyway), and Rabbit tries to do the most efficient thing rather than be FIFO in cases where it can.<div class="im">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The same problem occurs, when I get a set of messages and don't<br>
acknowledge them and close the channel (in the event of any exceptions).<br>
When we get the messages, they do not come in the correct order.<br>
</blockquote>
<br></div>
Exactly.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I guess, if i want to get sets of messages instead of one at a time, I<br>
must try to use the same channel. Is that right?<br>
</blockquote>
<br></div>
You could do that if it's convenient for you. Alternatively, you could prevent messages from being redelivered by making sure the server only sends you as many as you will actually process (if you know how many that is).<br>
<br>
Oleg suggests you set the prefetch count to 1 - this will ensure that the server only sends you one message until you ack it (which will block if you're trying to process 100 at once).<br>
<br>
So if you're currently reading in 100 messages and then acking all of them you could set the prefetch count to 100. The server will then allow you 100 unacked messages. You'd need to make sure you cancel before acking, otherwise the server can send you more messages between acking and closing the channel.<br>
<br>
So to summarise:<br>
<br>
channel.open<br>
basic.qos(prefetch_count = 100)<br>
basic.consume<br>
(consume 100 messages)<br>
basic.cancel<br>
(ack 100 messages)<br>
<br>
I'd be interested to know why you need to process messages in batches though.<br>
<br>
Cheers, Simon<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
<br>
On Thu, Jun 10, 2010 at 9:38 AM, Oleg Zhurakousky<br></div><div><div></div><div class="h5">
<<a href="mailto:ozhurakousky@vmware.com" target="_blank">ozhurakousky@vmware.com</a> <mailto:<a href="mailto:ozhurakousky@vmware.com" target="_blank">ozhurakousky@vmware.com</a>>> wrote:<br>
<br>
Not sure about C# API, and how you are building the next set of<br>
messages, but here is what I belive is happening.<br>
<br>
Channel prefetch messages. That is different then invoking a<br>
consumer. THis means that you might have 1000 prefetched messages in<br>
your channel, while you are still processing 21st message. Whatever<br>
you un-acknowledge is put back into the queue once the channel is<br>
closed. AMQO does not define an ordering of messages for re-queueing.<br>
One way to leverage this scenario is to set prefetchCount to 1. This<br>
way there will never me more then one prefetched messages in your<br>
channel.<br>
<br>
Oleg<br>
<br>
On Jun 10, 2010, at 10:24 AM, Srijanani Srinivasan wrote:<br>
<br>
<br>
Hi,<br>
<br>
I am using Rabbit MQ in C#. This is my scenario<br>
1. A separate process publishes messages to the queue<br>
2. Client has to read set of N messages from queue<br>
3. Process the N messages<br>
4. Acknowledge the N messages<br>
5. Repeat steps 2 to 4 continuously to process all<br>
sets of messages<br>
Under the same channel, I receive the messages and then process them<br>
and then acknowledge them. The server process keeps publishing<br>
messages. The problem I am facing is, when I try to get next set of<br>
messages, they do not come in the same order as it was published by<br>
the publishing process. The messages come in a random order. Only<br>
the first set of messages comes in the correct order.<br>
<br>
Does any one what is going wrong here? Is creating a new channel to<br>
access the next set of messages not right? Or is there a problem<br>
caused because of acknowledging multiple messages? Please help me<br>
understand why this does not work correctly.<br>
<br>
Below is the sample code:<br>
while (true)<br>
{<br>
using (IModel getChannel = MQConnection.CreateModel())<br>
{<br>
// Create a consumer<br>
QueueingBasicConsumer consumer =<br>
CreateQueueConsumer(getChannel, exchangeName, queueName);<br>
int numberOfMessages = 100;<br>
// Next Recieve<br>
List<object> msgSet =<br>
GetNextSetOfMessages(consumer, getChannel, exchangeName, queueName,<br>
numberOfMessages, out finalDeliverytag);<br>
// Do some processing<br>
//Acknowledge finished messages by passing in<br>
the delivery tag.<br>
// calls the method BasicAck with multiple<br>
param=true<br>
if (finalDeliverytag > 0)<br>
AckFinishedMessages(exchangeName, queueName,<br>
finalDeliverytag, getChannel);<br>
if (finalDeliverytag == 0)<br>
break;<br>
}<br>
}<br>
<br>
Thanks for your help in advance!<br>
Srijanani<br>
<br>
<ATT00001..txt><br>
<br>
<br>
<br>
<br></div></div>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</blockquote>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</blockquote></div><br>