[rabbitmq-discuss] Channels closed unexpectedly in Java client
Iain Hull
iain.hull at workday.com
Tue Sep 13 16:43:41 BST 2011
Hi,
I am currently running a stress test on one of our RabbitMQ production
servers. The goal of the test is to discover how many queues RabbitMQ
can support this will tell us how many tenants we can scale to on a
single box. The test creates a large number of queues, then for each
queue adds three tasks to a thread pool (with 20 threads). One task
writes 100 messages, the next reads 100 messages and the third task
reads and writes 100 messages. The messages are read with basicGet. We
are currently using RabbitMQ 2.5.1 server and Java client on CentOS
64bit with 8 cores and 96Gb of Ram (the high water mark is set to 0.8).
As we ramp up the number of queues above 32,000 some of the tasks start
to fail in basicGet with an IOException, there is also an error in the
server log file. As the queues increase these errors become more
frequent.
Queues: 32000 task success: 95997 errors: 3
Queues: 64000 task success: 159997 errors: 32003
Queues: 128000 task success: 287997 errors: 96003
The errors occur in batches and then clear up, however the numbers for
the errors are very suspicious as they are (n - 32000 + 3). During the
whole test only 6-8Gb of ram is used and rabbit server uses at most 350%
cpu (of 800%). I am wondering if this is a sign that I am hitting the
limit in the number of queues, I have miss configured RabbitMQ or my
test is not valid. Any help to figure this out would be appreciated.
Regards,
Iain.
=ERROR REPORT==== 13-Sep-2011::09:09:41 ===
connection <0.9484.159>, channel 1 - error:
{amqp_error,not_found,"no queue 'TestExchange_31997' in vhost '/'",
'basic.get'}
=ERROR REPORT==== 13-Sep-2011::09:09:41 ===
** Generic server <0.9652.183> terminating
** Last message in was {deliver,
{delivery,true,false,none,<0.9651.183>,
{basic_message,
{resource,<<"/">>,exchange,<<"TestExchange_31998">>},
[<<>>],
{content,60,
{'P_basic',<<"text/plain">>,undefined,undefined,2,
0,undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,undefined},
<<152,0,10,116,101,120,116,47,112,108,97,105,110,2,0>>,
rabbit_framing_amqp_0_9_1,
[<<"Hello World!0 from q=hello">>]},
<<18,156,101,170,65,11,23,81,252,168,116,117,202,153,
120,220>>,
true},
undefined}}
** When Server state == {q,
{amqqueue,
{resource,<<"/">>,queue,<<"TestExchange_31998">>},
true,false,none,[],<0.9652.183>},
none,false,rabbit_variable_queue,
{vqstate,
{[],[]},
{0,{[],[]}},
{delta,undefined,0,undefined},
{0,{[],[]}},
{[],[]},
0,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
undefined,
{0,nil},
{qistate,
"/var/lib/rabbitmq/mnesia/rabbit at dev182/queues/3N66B3H9OM1Y8OASL0S3XK8TT
",
{{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
[]},
undefined,0,262144,
#Fun<rabbit_variable_queue.2.111108431>,[]},
{{client_msstate,msg_store_persistent,
<<81,127,46,195,11,139,99,60,19,29,15,68,36,202,
246,12>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,245835,
"/var/lib/rabbitmq/mnesia/rabbit at dev182/msg_store_persistent"},
rabbit_msg_store_ets_index,
"/var/lib/rabbitmq/mnesia/rabbit at dev182/msg_store_persistent",
<0.240.0>,249932,241738,254029},
{client_msstate,msg_store_transient,
<<201,227,238,162,120,244,196,219,223,246,129,3,
224,211,0,204>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,229447,
"/var/lib/rabbitmq/mnesia/rabbit at dev182/msg_store_transient"},
rabbit_msg_store_ets_index,
"/var/lib/rabbitmq/mnesia/rabbit at dev182/msg_store_transient",
<0.232.0>,233544,225350,237641}},
{sync,[],[],[],[]},
true,0,#Fun<rabbit_amqqueue_process.4.73542412>,
#Fun<rabbit_amqqueue_process.5.110277333>,0,0,
infinity,0,0,0,0,0,0,
{rates,
{{1315,904981,880372},0},
{{1315,904981,880372},0},
0.0,0.0,
{1315,904981,880372}},
{0,nil},
{0,nil},
{0,nil},
{0,nil},
0,0,
{rates,
{{1315,904981,880372},0},
{{1315,904981,880372},0},
0.0,0.0,
{1315,904981,880372}}},
{[],[]},
{[],[]},
undefined,undefined,
{1315904986880394,#Ref<0.0.298.230323>},
undefined,
{state,fine,{1315904986880410,#Ref<0.0.298.230325>}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
undefined,undefined}
** Reason for termination ==
** {{badmatch,{error,emlink}},
[{rabbit_queue_index,get_journal_handle,1},
{rabbit_queue_index,publish,5},
{rabbit_variable_queue,maybe_write_index_to_disk,3},
{rabbit_variable_queue,maybe_write_to_disk,4},
{rabbit_variable_queue,publish,5},
{rabbit_variable_queue,publish,4},
{rabbit_amqqueue_process,deliver_or_enqueue,2},
{rabbit_amqqueue_process,handle_call,3}]}
Tue Sep 13 09:09:41 UTC 2011: java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:110)
at
com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:134)
at com.rabbitmq.client.impl.ChannelN.basicGet(ChannelN.java:806)
at
com.workday.messaging.test.BreakRabbitBreakTest.receiveMessages(BreakRab
bitBreakTest.java:200)
at
com.workday.messaging.test.BreakRabbitBreakTest.access$2(BreakRabbitBrea
kTest.java:198)
at
com.workday.messaging.test.BreakRabbitBreakTest$SendAndReceive.perform(B
reakRabbitBreakTest.java:262)
at
com.workday.messaging.test.BreakRabbitBreakTest$AbstractMessageRunner.ca
ll(BreakRabbitBreakTest.java:234)
at
com.workday.messaging.test.BreakRabbitBreakTest$AbstractMessageRunner.ca
ll(BreakRabbitBreakTest.java:1)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecuto
r.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error;
reason: {#method<channel.close>(reply-code=404, reply-text=NOT_FOUND -
no queue 'TestExchange_31997' in vhost '/', class-id=60,
method-id=70),null,""}
at
com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
at
com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(Bl
ockingValueOrException.java:33)
at
com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQ
Channel.java:337)
at
com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:210)
at
com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:128)
... 14 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error;
reason: {#method<channel.close>(reply-code=404, reply-text=NOT_FOUND -
no queue 'TestExchange_31997' in vhost '/', class-id=60,
method-id=70),null,""}
at
com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:422)
at
com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:262)
at
com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChan
nel.java:154)
at
com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:99)
at
com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:4
43)
Tue Sep 13 09:09:41 UTC 2011: java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:110)
at
com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:134)
at com.rabbitmq.client.impl.ChannelN.basicGet(ChannelN.java:806)
at
com.workday.messaging.test.BreakRabbitBreakTest.receiveMessages(BreakRab
bitBreakTest.java:200)
at
com.workday.messaging.test.BreakRabbitBreakTest.access$2(BreakRabbitBrea
kTest.java:198)
at
com.workday.messaging.test.BreakRabbitBreakTest$SendAndReceive.perform(B
reakRabbitBreakTest.java:262)
at
com.workday.messaging.test.BreakRabbitBreakTest$AbstractMessageRunner.ca
ll(BreakRabbitBreakTest.java:234)
at
com.workday.messaging.test.BreakRabbitBreakTest$AbstractMessageRunner.ca
ll(BreakRabbitBreakTest.java:1)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecuto
r.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error;
reason: {#method<channel.close>(reply-code=404, reply-text=NOT_FOUND -
no queue 'TestExchange_31998' in vhost '/', class-id=60,
method-id=70),null,""}
at
com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
at
com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(Bl
ockingValueOrException.java:33)
at
com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQ
Channel.java:337)
at
com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:210)
at
com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:128)
... 14 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error;
reason: {#method<channel.close>(reply-code=404, reply-text=NOT_FOUND -
no queue 'TestExchange_31998' in vhost '/', class-id=60,
method-id=70),null,""}
at
com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:422)
at
com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:262)
at
com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChan
nel.java:154)
at
com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:99)
at
com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:4
43)
More information about the rabbitmq-discuss
mailing list