[rabbitmq-discuss] ShutdownSignalException second 'channel.open'
Steve Powell
steve at rabbitmq.com
Wed Feb 1 10:35:30 GMT 2012
Yogesh,
Thanks for your explanation of your application structure, and the version you
are running.
I do not know how the second channel.open AMQP command (for the same channel)
was sent to the broker; can you send some more diagnostics of this failure (log,
full stack trace) so I can raise a bug. If possible, a demonstrating small
program. Thanks.
I see your difficulty. I would hope that creating 20000 queues (which are empty
almost all of the time) would not take up too much room, so I think you should
consider not deleting them at all. This would solve most of your problem.
The x-expires attribute for queues, really ought to have a 'only-if-empty'
option. I'll raise a bug (24722) for this, and see how difficult it would be.
I do not understand what the _Return queues are for. They look as though you are
using them as a private persistent store for each App. It would be better if you
put this information somewhere else, but heigh-ho -- you can safely delete these
queues since this app is in complete control of them, and the events are
processed serially.
One of the reasons, I don't do basicConsume on queues is, I am going to have
thousands of queues. I rather thought it would be easier to have a thread pool,
each consuming just one message from a queue in round-robin fashion.
Just because you have thousands of queues doesn't mean you need to have
thousands of Consumer instances -- it is quite ok to use the same Consumer on
several basicConsume calls. The consumerTag is passed to every Consumer callback
so the Consumer code can distinguish for which basicConsume it is called.
By default, the Java client will allocate a pool of 5 threads for each
connection, and dispatch Consumer callbacks onto one of those threads for you.
So your thread pool might be unnecessary if you used basicConsume(). I'd
consider doing that since basicGet() can be slow, and you'd have to get from
each potential queue just to tell if there is a message for it!
Steve Powell (a happy bunny)
----------some more definitions from the SPD----------
vermin (v.) Treating the dachshund for roundworm.
chinchilla (n.) Cooling device for the lower jaw.
socialcast (n.) Someone to whom everyone is speaking but nobody likes.
On 31 Jan 2012, at 13:56, Yogesh Ketkar wrote:
> I am using RabbitMQ server version 2.7.1.
> Java Client Jar used is
> rabbitmq-java-client-bin-2.7.1/rabbitmq-client.jar
>
>> The Shutdown seems to be being called because the channel is being opened twice.
>> The broker complains about this and closes the connection. Are you creating
>> channels on different threads simultaneously?
> Yes indeed, I am creating channels on different threads.
>
>> I don't think the basicPublish will fail if the queue doesn't exist. Why would
>> you create a new channel in this case?
> Yes, you are right. I will basically lose the message in this case.
>
> Now about overall problem statement.
> My application has a main queue which looks like
> MainQueue
> - App1-Event1
> - App2-Event1
> - App1-Event2
> - App1-Event3
> - App3-Event1
> - App3-Event2
> - App2-Event2
>
> Basically there are going to be events from different Apps (there can
> be thousands of apps) and events belonging to an App must
> be processed sequentially. Events across different apps can and should
> be be processed in parallel.
> So I have only one consumer on MainQueue (using basicConsume) which
> reads events from MainQueue and just moves it to appropriate declared/
> redeclared queue.
> So this is how new queue structure would look like.
>
> App1
> - App1-Event1
> - App1-Event2
> - App1-Event3
>
> App2
> - App2-Event1
> - App2-Event2
>
> App3
> - App3-Event1
> - App3-Event2
>
>
> Now again when Event1 is processed from Queue App1, Event2 of App1
> can't be processed unless processing of Event1 is complete.
> Processing of event involves asynchronous communication with external
> systems, so once Event1 is fetched (and acknowledged) from queue
> App1,
> I create another queue like
> App1_Return
> - App1-Event1-TaskId
>
> I need to query external system using TaskId after certain time
> interval, to check status of event processing of Event1. Once I get
> the status (either sucess or failure)
> I discard App1-Event1-TaskId and ready to process App1-Event2. So all
> _Return queues will only have one event at any point of time.
>
> An event on an app might even occur once a day. So I don't want to
> keep so many queues (potentially 20000 if there are 10000 apps)
> hanging around.
> Both auto_delete and x-expires are not very useful as in both the
> schemes, queues get deleted even when they have messages.
> Ideally whenever last message from any Queue (except MainQueue) is
> consumed, I want to delete that queue. Of course, one has to make sure
> while a queue is getting deleted, there might be an event destined for
> that. So if one guy is doing
> queueDelete('somequeue', true, true) and other guy is doing
> queueDeclare, queueBind, basicPublish. If queueDelete gets executed
> after
> queueBind, message will be lost.
>
> One of the reasons, I don't do basicConsume on queues is, I am going
> to have thousands of queues. I rather thought it would be easier to
> have a thread pool, each consuming just one message from a queue in
> round-robin fashion.
>
> As was mentioned in some other response, I will certainly not create a
> new channel in every thread, but would rather try and reuse them.
>
>
> Thanks for all the help.
> Regards, Yogesh
>
>
>
> On Jan 31, 5:19 pm, Steve Powell <st... at rabbitmq.com> wrote:
>> Yogesh,
>>
>> Please can you provide some information about your environment? And your
>> application? What version of RabbitMQ (and client) are you using?
>>
>> In your stack trace the ShutdownListener you registered is apparently being
>> called, because the Connection is being shut down. It is not clear why this
>> exception (and its associated stack trace) appears, it seems to come from your
>> Listener code, but perhaps that does nothing.
>>
>> The Shutdown seems to be being called because the channel is being opened twice.
>> The broker complains about this and closes the connection. Are you creating
>> channels on different threads simultaneously? (Looking at your app 'design' you
>> might be.) Depending upon the version of RabbitMQ this might cause a problem.
>>
>> I'm afraid your application design is unclear:
>>
>>> This is how I handle doing basicPublish and basicGet on potentially
>>> non-existent queues
>>> - publish involves 3 steps
>>> queueDeclare
>>> queueBind
>>> basicPublish
>>> If some other thread deletes the queue after either queueDeclare or
>>> queueBind, basicPublish fails and I again create a new
>>> channel and do these operations
>>
>> I don't think the basicPublish will fail if the queue doesn't exist. Why would
>> you create a new channel in this case?
>>
>> Please explain why you expect the queue might be deleted by some other thread.
>>
>>> - if basicGet fails, I simply ignore it
>>
>> What do you mean by ignoring it? Do you poll the queue periodically? Why aren't
>> you using basicConsume and a Consumer to get messages (which will be notified if
>> the queue is deleted)?
>>
>> Steve Powell (a loopy bunny)
>> ----------some more definitions from the SPD----------
>> vermin (v.) Treating the dachshund for roundworm.
>> chinchilla (n.) Cooling device for the lower jaw.
>> socialcast (n.) Someone to whom everyone is speaking but nobody likes.
>>
>> On 30 Jan 2012, at 04:33, Yogesh Ketkar wrote:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>> Only operations I ever do with com.rabbitmq.client.Connection in the
>>> code are
>>> c.addShutdownListener
>>> c.createChannel
>>
>>> What does this error signify?
>>
>>> 2012-01-30 09:44:45,158 ERROR [ConnectionShutdownHandler]
>>> ShutdownListener
>>> com.rabbitmq.client.ShutdownSignalException: connection error; reason:
>>> {#method<connection.close>(reply-code=503, reply-text=COMMAND_INVALID
>>> - second 'channel.open' seen, class-id=20, method-id=10), null,
>>> "[B at 105691e"}
>>> at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:
>>> 641)
>>> at
>>> com.rabbitmq.client.impl.AMQConnection.handleConnectionClose(AMQConnection. java:
>>> 599)
>>> at
>>> com.rabbitmq.client.impl.AMQConnection.processControlCommand(AMQConnection. java:
>>> 571)
>>> at com.rabbitmq.client.impl.AMQConnection
>>> $1.processAsync(AMQConnection.java:88)
>>> at
>>> com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel .java:
>>> 144)
>>> at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:
>>> 91)
>>> at com.rabbitmq.client.impl.AMQConnection
>>> $MainLoop.run(AMQConnection.java:500)
>>
>>> Some additional info.
>>> I create and close thousands of channels in the code. But at any point
>>> of time there are not more than 20/21 channels open.
>>> This is how I handle doing basicPublish and basicGet on potentially
>>> non-existent queues
>>> - publish involves 3 steps
>>> queueDeclare
>>> queueBind
>>> basicPublish
>>> If some other thread deletes the queue after either queueDeclare or
>>> queueBind, basicPublish fails and I again create a new
>>> channel and do these operations
>>> - if basicGet fails, I simply ignore it
>>
>>> regards, Yogesh
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-disc... at lists.rabbitmq.com
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
More information about the rabbitmq-discuss
mailing list