[rabbitmq-discuss] Sequential message processing guarantee
Steve Powell
steve at rabbitmq.com
Wed Jan 25 14:42:53 GMT 2012
Not only more appropriate, but also more true :-)
Steve Powell (a happy bunny)
----------some more definitions from the SPD----------
vermin (v.) Treating the dachshund for roundworm.
chinchilla (n.) Cooling device for the lower jaw.
socialcast (n.) Someone to whom everyone is speaking but nobody likes.
On 24 Jan 2012, at 05:32, Yogesh Ketkar wrote:
>> Well, having more threads will help if same channel is going to be used to
>> consume messages from other queues.
> Actually more appropriate would be, having more threads will help if
> same Connection is going to be used
> with different channels.
>
> On Jan 24, 10:29 am, Yogesh Ketkar <yogim... at gmail.com> wrote:
>> Hello Steve/Alex,
>>
>> Thanks a lot for the help. It's pretty clear now.
>> So basically for a queue, as long as there is going to be only one
>> consumer, it hardly matters if ThreadPool with
>> com.rabbitmq.client.Connection underneath the channel has 1 thread or many
>> threads.
>> Well, having more threads will help if same channel is going to be used to
>> consume messages from other queues.
>>
>> regards, Yogesh
>>
>> 2012/1/23 Steve Powell <st... at rabbitmq.com>
>>
>>
>>
>>
>>
>>
>>
>>> Hi Yogesh,
>>
>>> What Alex says is perfectly correct.
>>
>>> The executor service is set on the connection, and is there to allow
>>> multiple
>>> channels to execute consumers simultaneously, and without blocking the
>>> channel
>>> thread. By default you get an executor service which has five threads.
>>
>>> From the point of view of a single channel, the consumers are driven
>>> serially,
>>> one delivery at a time, upon one of the executor service threads. The same
>>> thread is not guaranteed to be used each time for each consumer call for a
>>> channel (so don't try to use ThreadLocal objects).
>>
>>> The reason we provide this, is so that clients with many, many channels
>>> are not
>>> forced to have a Consumer thread per channel -- something that is quite
>>> expensive -- but have the option of allocating as many threads as they
>>> wish if
>>> they need it. At the same time we guarantee that no Consumer calls can
>>> 'overtake' each other on any one channel, and that Consumer processing
>>> runs on a
>>> distinct thread from the channel and the app, so that channel calls in the
>>> Consumer code will not deadlock.
>>
>>> I hope this clarifies rather than obscures the situation,
>>
>>> Regards,
>>> Steve Powell (*a hoppy bunny*)
>>> *----------some more definitions from the SPD----------*
>>> *vermin* (*v.*) Treating the dachshund for roundworm.
>>> *chinchilla* (*n.*) Cooling device for the lower jaw.
>>> *socialcast* (*n.*) Someone to whom everyone is speaking but nobody likes.
>>
>>> On 23 Jan 2012, at 15:22, Alexandru Scvorţov wrote:
>>
>>> Is my understanding correct?
>>
>>> Almost. Deliveries will happen serially on a channel, so, as long as you
>>> only have one consumer, everything should be fine.
>>
>>> On Mon, Jan 23, 2012 at 06:54:15AM -0800, Yogesh Ketkar wrote:
>>
>>> Hello Alex,
>>
>>> I thought, using default Connection con = factory.newConnection(),
>>
>>> ThreadPool which gets created will have more than one thread and
>>
>>> calling basicConsume will result in processing of messages on multiple
>>
>>> threads.
>>
>>> But if we explicitly provide Executors.newSingleThreadExecutor() to
>>
>>> newConnection method, only single thread will come into play with
>>
>>> basicConsume.
>>
>>> Is my understanding correct?
>>
>>> regards, Yogesh
>>
>>> On Jan 23, 7:48 pm, Alexandru Scvorţov <alexan... at rabbitmq.com> wrote:
>>
>>> Hi Yogesh,
>>
>>> using channel.basicConsume on the channel which is created like this
>>
>>> ExecutorService es = Executors.newSingleThreadExecutor();
>>
>>> Connection connection = factory.newConnection(es);
>>
>>> final Channel channel = connection.createChannel();
>>
>>> is probably a better approach.
>>
>>> Any comments on this?
>>
>>> I don't quite see the point of the ExecutorService.
>>
>>> Just doing this:
>>
>>> Connection connection = factory.newConnection(es);
>>
>>> final Channel channel = connection.createChannel();
>>
>>> Then, calling basicConsume with your own subclass of DefaultConsumer
>>
>>> should be fine.
>>
>>> Cheers,
>>
>>> Alex
>>
>>> On Mon, Jan 23, 2012 at 08:08:37PM +0530, Yogesh Ketkar wrote:
>>
>>> Thanks Alex.
>>
>>> I think, for sequential message processing, rather than this pattern
>>
>>> while(true) {
>>
>>> GetResponse res = channel.basicGet(QUEUE_NAME, false);
>>
>>> if(res != null) {
>>
>>> // process and ack message
>>
>>> }
>>
>>> }
>>
>>> using channel.basicConsume on the channel which is created like this
>>
>>> ExecutorService es = Executors.newSingleThreadExecutor();
>>
>>> Connection connection = factory.newConnection(es);
>>
>>> final Channel channel = connection.createChannel();
>>
>>> is probably a better approach.
>>
>>> Any comments on this?
>>
>>> regards, Yogesh
>>
>>> 2012/1/23 Alexandru Scvorţov <alexan... at rabbitmq.com>
>>
>>> Hi Yogesh,
>>
>>> In the absence of consumer failures, RabbitMQ will deliver messages from
>>
>>> a queue in order.
>>
>>> So, if messages 1, 2, 3 reach a queue in order, RabbitMQ will deliver
>>
>>> them to consumers in the same order (1, 2, 3).
>>
>>> But, if consumers fail before acknowledging the messages (or if they
>>
>>> reject the messages), those messages will requeued at the end of the
>>
>>> queue. So, if the consumer that got message 1 rejects it, the new
>>
>>> order of messages will be 2, 3, 1.
>>
>>> This all deals with message *delivery*.
>>
>>> There are multiple consumer threads consuming the messages.
>>
>>> Though multiple threads are consuming the messages, I observed that
>>
>>> messages still get processed sequentially.
>>
>>> Is that the case and if NOT, what is the way to guarantee sequential
>>
>>> processing of messages on one queue?
>>
>>> The library makes no guarantees about the order in which you process
>>
>>> messages, only about the order in which they're delivered.
>>
>>> If you want to process all the messages on a queue in order, only
>>
>>> consume from one thread (doing basic.get from one thread like you
>>
>>> suggested in the other email would work, but would also be highly
>>
>>> inefficient).
>>
>>> Does this answer your question?
>>
>>> Cheers,
>>
>>> Alex
>>
>>> On Sun, Jan 22, 2012 at 09:38:40AM -0800, Yogesh Ketkar wrote:
>>
>>> Running the code below, gives
>>
>>> channel.basicConsume(QUEUE_NAME, autoAck, CONSUMER_TAG,
>>
>>> new DefaultConsumer(channel) {
>>
>>> @Override
>>
>>> public void handleDelivery(String consumerTag, Envelope
>>
>>> envelope, BasicProperties properties, byte[] body) {
>>
>>> System.out.println(Thread.currentThread().getName());
>>
>>> }
>>
>>> }
>>
>>> o/p like
>>
>>> pool-1-thread-1
>>
>>> pool-1-thread-2
>>
>>> pool-1-thread-3
>>
>>> etc
>>
>>> There are multiple consumer threads consuming the messages.
>>
>>> Though multiple threads are consuming the messages, I observed that
>>
>>> messages still get processed sequentially.
>>
>>> Is that the case and if NOT, what is the way to guarantee sequential
>>
>>> processing of messages on one queue?
>>
>>> regards, Yogesh
>>
>>> _______________________________________________
>>
>>> rabbitmq-discuss mailing list
>>
>>> rabbitmq-disc... at lists.rabbitmq.com
>>
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>> _______________________________________________
>>
>>> rabbitmq-discuss mailing list
>>
>>> rabbitmq-disc... at lists.rabbitmq.comhttps://
>>> lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>> _______________________________________________
>>
>>> rabbitmq-discuss mailing list
>>
>>> rabbitmq-disc... at lists.rabbitmq.com
>>
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-disc... at lists.rabbitmq.com
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120125/cec3dc74/attachment.htm>
More information about the rabbitmq-discuss
mailing list