[rabbitmq-discuss] RabbitMQ - Not able to get sets of messages in the same order
Simon MacMullen
simon at rabbitmq.com
Tue Jun 15 18:28:21 BST 2010
Hi Srijanani.
On 15/06/10 17:45, Srijanani Srinivasan wrote:
> Thanks for your response Simon.
>
> I was not using Basic.QOS method to set the prefetch count so far. After
> your suggestion, I modified the code to follow the same order like you
> said and I tested it. I see that the order is preserved, when a new
> channel/consumer is created for accessing the next set of messages. But,
> if we don't acknowledge a certain set of messages, like you said, the
> order is not preserved as they will be getting re-delivered.
>
> In my application, we have high volume of messages that are going to be
> coming in through the queue. And we will have to do some calculation and
> store data in database. We have realized earlier that saving the data
> one by one is time consuming, so we will take one set of say
> 5000 messages and then do the calculations and store it the database in
> one database call. So basically, we do not want to process one by one to
> save database calls. Another reason, is that there could be a lot of
> duplicate messages, so we will pick 5000 and do the work only for the
> unique ones.
Ah, this makes sense.
> Is there any performance gain if we ACK a set of messages instead of
> ACKing one by one, as we are reducing the number of responses sent from
> the client side when we do bulk ACks?
Yes, as long as you use the multiple flag on basic.ack. This allows you
to acknowledge all messages up to and including the delivery tag you
give, and only one ack message is sent over the wire.
Cheers, Simon
>
> On Tue, Jun 15, 2010 at 7:45 AM, Simon MacMullen <simon at rabbitmq.com
> <mailto:simon at rabbitmq.com>> wrote:
>
> Hi Srijanani.
>
>
> On 10/06/10 15:53, Srijanani Srinivasan wrote:
>
> Thanks for your response. What you said makes sense. When the
> messages
> that have not been acknowledged are put back to the queue, then
> the other messages added in the meantime should be the ones to be
> de-queued first right in the FIFO manner? But like you said, i guess
> thats not how it works.
>
>
> There isn't a guarantee about ordering for redelivered messages in
> the spec (since in more complex cases it's impossible to guarantee
> anyway), and Rabbit tries to do the most efficient thing rather than
> be FIFO in cases where it can.
>
>
> The same problem occurs, when I get a set of messages and don't
> acknowledge them and close the channel (in the event of any
> exceptions).
> When we get the messages, they do not come in the correct order.
>
>
> Exactly.
>
>
> I guess, if i want to get sets of messages instead of one at a
> time, I
> must try to use the same channel. Is that right?
>
>
> You could do that if it's convenient for you. Alternatively, you
> could prevent messages from being redelivered by making sure the
> server only sends you as many as you will actually process (if you
> know how many that is).
>
> Oleg suggests you set the prefetch count to 1 - this will ensure
> that the server only sends you one message until you ack it (which
> will block if you're trying to process 100 at once).
>
> So if you're currently reading in 100 messages and then acking all
> of them you could set the prefetch count to 100. The server will
> then allow you 100 unacked messages. You'd need to make sure you
> cancel before acking, otherwise the server can send you more
> messages between acking and closing the channel.
>
> So to summarise:
>
> channel.open
> basic.qos(prefetch_count = 100)
> basic.consume
> (consume 100 messages)
> basic.cancel
> (ack 100 messages)
>
> I'd be interested to know why you need to process messages in
> batches though.
>
> Cheers, Simon
>
>
> On Thu, Jun 10, 2010 at 9:38 AM, Oleg Zhurakousky
> <ozhurakousky at vmware.com <mailto:ozhurakousky at vmware.com>
> <mailto:ozhurakousky at vmware.com
> <mailto:ozhurakousky at vmware.com>>> wrote:
>
> Not sure about C# API, and how you are building the next set of
> messages, but here is what I belive is happening.
>
> Channel prefetch messages. That is different then invoking a
> consumer. THis means that you might have 1000 prefetched
> messages in
> your channel, while you are still processing 21st message.
> Whatever
> you un-acknowledge is put back into the queue once the
> channel is
> closed. AMQO does not define an ordering of messages for
> re-queueing.
> One way to leverage this scenario is to set prefetchCount to
> 1. This
> way there will never me more then one prefetched messages in
> your
> channel.
>
> Oleg
>
> On Jun 10, 2010, at 10:24 AM, Srijanani Srinivasan wrote:
>
>
> Hi,
>
> I am using Rabbit MQ in C#. This is my scenario
> 1. A separate process publishes messages to
> the queue
> 2. Client has to read set of N messages from queue
> 3. Process the N messages
> 4. Acknowledge the N messages
> 5. Repeat steps 2 to 4 continuously to process all
> sets of messages
> Under the same channel, I receive the messages and then
> process them
> and then acknowledge them. The server process keeps publishing
> messages. The problem I am facing is, when I try to get next
> set of
> messages, they do not come in the same order as it was
> published by
> the publishing process. The messages come in a random order.
> Only
> the first set of messages comes in the correct order.
>
> Does any one what is going wrong here? Is creating a new
> channel to
> access the next set of messages not right? Or is there a problem
> caused because of acknowledging multiple messages? Please
> help me
> understand why this does not work correctly.
>
> Below is the sample code:
> while (true)
> {
> using (IModel getChannel =
> MQConnection.CreateModel())
> {
> // Create a consumer
> QueueingBasicConsumer consumer =
> CreateQueueConsumer(getChannel, exchangeName, queueName);
> int numberOfMessages = 100;
> // Next Recieve
> List<object> msgSet =
> GetNextSetOfMessages(consumer, getChannel, exchangeName,
> queueName,
> numberOfMessages, out finalDeliverytag);
> // Do some processing
> //Acknowledge finished messages by
> passing in
> the delivery tag.
> // calls the method BasicAck with multiple
> param=true
> if (finalDeliverytag > 0)
> AckFinishedMessages(exchangeName,
> queueName,
> finalDeliverytag, getChannel);
> if (finalDeliverytag == 0)
> break;
> }
> }
>
> Thanks for your help in advance!
> Srijanani
>
> <ATT00001..txt>
>
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> <mailto:rabbitmq-discuss at lists.rabbitmq.com>
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> <mailto:rabbitmq-discuss at lists.rabbitmq.com>
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
More information about the rabbitmq-discuss
mailing list