<div dir="ltr">Cheers!<div><br></div><div> I'm in the process of building a rabbitmq cluster, and before going to production I'd like to produce some benchmarks, and run stresstests.</div><div><br></div><div> Unfortunately rabitmq 3.1.3 (stable ubuntu 13.10), and 3.2.2 (stable ubuntu 13.10 + <a href="http://rabbitmq.com">rabbitmq.com</a> repository) crashes miserably every time.</div>
<div><br></div><div><br></div><div> I've got 3 nodes, called node1-2-3, but don't get confused, node1 is for stresstesting only, and node2+node3 are in a rabbitmq cluster.</div><div><br></div><div> All nodes have 16G of ram and about 100G of disk space. ERLANG16B1. Hipe is installed, but not enabled in the configuration as it's marked as experimental, and I'd like to get stability first, performance is secondary, although that would be nice too. :)</div>
<div><br></div><div> For benchmarking I'd do a lot of test, most of them do not fail, so we will skip those for now. :)</div><div><br></div><div> I run perftest to "attack" the cluster like this:</div><div>
<br>
</div><div> consumer:
/perftest/rabbitmq-java-client-bin-3.2.2//runjava.sh
com.rabbitmq.examples.PerfTest -h
'amqp://benchmark:xxx@node3/benchmark' --consumers 5
--producers 0 -u benchmarkq -p --cmessages 10000 --multiAckEvery 100 2>&1 <br></div><div><br></div><div><br></div><div>publisher:
perftest/rabbitmq-java-client-bin-3.2.2//runjava.sh
com.rabbitmq.examples.PerfTest -h
'amqp://benchmark:xxx@node2/benchmark' --consumers 0
--producers 200 --routingKey 'a.FOOBAR.c' --exchange 'benchmark.topic'
-p -u benchmarkq --size 5242880 --pmessages 50 2>&1 <br></div><div><br></div><div> So from node1 I publish 10000 messages (5mbytes each) to a queue on node2, and simultaneously I consume 10000 messages through node3. (The client is on node1, which is still NOT PART of the rabbitmq cluster, it just has a name like node1)</div>
<div><br></div><div> For the tests I use a HA policy to duplicate the messages on both nodes, but it should not make a big difference in network transfer, as the consumer connects to node3, so everything (50Gbyte of data) must go around, but it surely makes the things slower as node3 has to write everything to disk.</div>
<div><br></div><div> The nodes have both management plugin enabled, but have no special configuration, so no shovel no federation, they are disk nodes, high memory watermark set to 0.4.</div><div><br></div><div>-------------------------</div>
<div> Crash type 1.</div><div><br></div><div> Previously I've reported this bug on irc to bob235, who got the logs for that crash and responded as it's a real bug in the software, and asked me to write down the specifics, so here we are.</div>
<div><br></div><div> He acknowledged this as a real bug </div><div> ("{gm,find_prefix_common_suffix,2,[]}" in the stacktrace)</div><div> </div><div> (trace can be found at: <a href="http://corpweb.dunakanyar.net/petrosdump/rabbitcrashlogs.tgz">http://corpweb.dunakanyar.net/petrosdump/rabbitcrashlogs.tgz</a> )</div>
<div><br></div><div> These logs shows the death of a node, and the sasl log has the details... The "attack" was like the above, 200 producers.</div><div><br></div><div> One thing to note is that the network links were unbalanced:</div>
<div> - node1->node2 1000mbit</div><div> - node2->node3 100mbit</div><div> - node3->node1 100mbit</div><div><br></div><div> But this is not a reason for rabbitmq to stop working. :)</div><div><br></div><div>-------------------------</div>
<div> Crash type 2.</div><div><br></div><div> Situation: as the consumers have a slower network link, disk runs out on node2, and while I had the default setting of (I think 50mbytes), rabbitmq run out of disk space, and CRASHED!. </div>
<div><br></div><div> I don't think high availability has a part of "if (last_err==E_NOSPC) die();" or something, so rabbitmq should work like it's documented: <a href="http://www.rabbitmq.com/memory.html">http://www.rabbitmq.com/memory.html</a></div>
<div><br></div><div> Flow control should kick in whenever a disk capacity problem is detected, and publishers should stop for a while, and consumers will help the server to breath...</div><div><br></div><div> 2.1: If the server does not work like it's documented, then it's a bug.</div>
<div> 2.2: If the server's default free space limit (50mybte) does not make running a stable service, then it's not a good default setting</div><div> 2.3 If I have 50gbyte of total data (10000 pcs of 5mbyte messages) in a single queue (and It's even consumed while published, so there's never a moment where every message is on the server), I don't believe 90gbyte of disk space should be allocated, and the server should crash. This is not a documented feature I think, so maybe the documentation needs to get extended.</div>
<div> 2.4 Could we get a documentation in a backward accessible way, so for example <a href="http://rabbitmq.com/doc/3.1.8/flowcontrol.html">rabbitmq.com/doc/3.1.8/flowcontrol.html</a> ? This would make things easier when someone does not use the latest.</div>
<div> 2.5 (as a side note to default settings: guest:guest as admin, default open to everyone may not be the best setting too, as users tend to leave this enabled, then fail a few months later) :) )</div><div><br></div>
<div>
So the flow control does not work well for out of disk space situations, even if the amount of data never reaches the disk space available, the server can crash, and this is not nice, the server is not failsafe.</div><div>
<br></div><div>--------------------</div><div>Crash type 3.</div><div><br></div><div> Situation: Out of memory.</div><div> </div><div> I've got 16Gbytes of ram in each node. The watermark is set to 7.8Gbyte, so we should never have memory problems.</div>
<div><br></div><div> If I start the perftest with 10 publishers (1000messages x 5mbytes), and 5 consumers, the rabbitmq server keeps allocating ram.</div><div><br></div><div> 3.1 Management plugin tells me that it's using for example 9 gigabytes (HOW!?), but in real the os shows me, that it's over 12gbytes. </div>
<div><br></div><div> As the perftest is on the way, the server keeps allocating more and more memory. We're swapping now, as rabbitmq is using over 20gigabytes of memory. (Still has the watermark at 7.8!)</div><div><br>
</div><div> And at this moment the lighning strikes: </div><div><br></div><div><div>cat /var/log/rabbitmq/startup_err </div><div><br></div><div>"Crash dump was written to: erl_crash.dump</div><div>eheap_alloc: Cannot allocate 8162366936 bytes of memory (of type "old_heap").</div>
<div>Aborted (core dumped)"</div></div><div> </div><div> WTF (What a terrible failure - after Android SDK :D) </div><div><br></div><div> 3.2 So the server configured to use 7.8gbytes memory max tried to allocate 8 gigabytes over the already allocated 20gigabytes. This is surely a bug.</div>
<div><br></div><div> 3.3 flow control kicked in, as the logs show, but memory usage did not lower... and running out of memory is not something that is out of control and a failsafe server "could not handle" like a kill -9. It is a bug to let it go over 3 times the allowed memory limit.</div>
<div> 3.4 and it's an another bug not to handle the E_NOMEM situation. </div><div> 3.5 writing a 19gigabyte erl_crash.dump takes a lot of time :), and it's not even a binary memorydump with all the data, it's just the stacktrace :D</div>
<div><br></div><div> The full logs are here: </div><div><br></div><div><a href="http://corpweb.dunakanyar.net/petrosdump/rabbitmqoutofmemory20140113.tar.bz2">http://corpweb.dunakanyar.net/petrosdump/rabbitmqoutofmemory20140113.tar.bz2</a> (1.3gigabytes, contains the full 19gbyte stacktrace too, watch out when uncompressing)<br>
</div><div><br></div><div> I'll remove this file at 2014-02-01.</div><div><br></div><div>(I know the db contains the users, passes, but they are worthless as it's just a drop-rebuild machine, even the cookie means nothing)</div>
<div><br></div><div> <br></div><div>-------</div><div>Crash type 4.</div><div> This happened a few times too, even after fixing the 100m network bottleneck between node3 and others.</div><div><br></div><div> Sometimes the server goes down with the message </div>
<div><br></div><div>"Absurdly large distribution output data buffer (2427628186 bytes) passed.</div><div>Aborted (core dumped)"</div><div><br></div><div> I can't find an erl_crash.dump in this case.</div><div>
<br></div><div> But I have a log for this too: <a href="http://corpweb.dunakanyar.net/petrosdump/cluster_node2_20140109_163509.tar.bz2">http://corpweb.dunakanyar.net/petrosdump/cluster_node2_20140109_163509.tar.bz2</a></div>
<div><br></div><div> The test was something similar, probably 200 publishers * 50 messages each 5mbytes. </div><div><br></div><div> So total data amount is always 50gigabytes, the total message count is 10.000, I just play with the producer count sometimes.</div>
<div><br></div><div><br></div><div>--- </div><div><br></div><div> Questions:</div><div><br></div><div> - What am I doing wrong? Shouldn't be this "10000 messages from publishers to consumers" a simple usecase? Why does the message size make difference? (This is just the size of my local mailbox, and thunderbird does not crash. :) )</div>
<div> - How could I help to determine and remove the factors that keep rabbitmq crashing?</div><div> - What can I do to deliver 50 pcs of message from each of 200 workers?</div><div> - Which is the "pivotal preferred stable version"? And which is the "pivotal preferred stable linux distribution"?</div>
<div> - Does rabbitmq have a ticket system somewhere?</div><div> - Where to report bugs to?</div><div><br></div><div><br></div><div> I can reproduce some of these, but it looks random, which problem kills the server faster. :)</div>
<div><br></div><div> Of course I've raised the disk free limit, so I don't hit that problem today.</div><div><br></div><div> As these are testing machines currently it is possible to give a developer access to them if it helps.</div>
<div><br></div><div> Please help me get a failsafe rabbitmqnode to build a failsafe rabbitmq cluster from. :)</div><div><br></div><div> Any ideas?</div><div><br></div><div> Thank you!</div><div><br></div><div> Peter Kopias</div>
<div><br></div><div>ps.: I started with 3.1.3, but I figured that reporing crashes in a year old release does not interest the devs, so I switched to 3.2.2, as it shows a lot of crashes fixed in the changelog. All the above happened to 3.2.2 serverd by <a href="http://rabbitmq.com">rabbitmq.com</a> ...</div>
<div><br></div></div>