Hi all<br><br>First I need to send kudos to all of you, I've been using rabbit for a year now and has given me a lot of satisfactions. It reminded me the day I use a Database instead of a bunch of files.. the same but with data flows! Like a marriage, satisfactions and problems come in the same pack! Just some background info: <br>
<br> * Im running rabbitmq 1.7.0-3 on a debian testing distribution with python-amqplib 0.6.1-1<br> * It's a VPS with 2GB of RAM, *BUT* when the used memory it's aboout 80%, "tasks may be killed off pre-maturely, we do this to prevent a total OOM scenario where the VPS goes dead" (literal words from the provider)<br>
* I was using ~15 queues with 1kb messages and a rate of ~20 messages/minute <br><br>Like a week ago I've made the following changes: <br> * I'm using 50 queues with a rate of ~60 messages/minute <br> * One of them (I call it "the fat queue" ) has 50kb messages<br>
* Processing at the rear end is rate fixed ... so I need rabbit to act as a memory/disk buffer <br><br>A: High speed producer : 50000 1K messages in a second or so.<br>B: Translator: reads from A_q and generates a 50kb message from a single 1kb message and publishes it on the fat_queue.<br>
C: (Slow) Fixed speed consumer: grabs the 50kb message and does it's magic.<br><br>A process -> A_q queue -> B process -> fat_queue -> many C processes (even that I have many Cs, A is much faster) <br><br>
<br>Here is the problem when everythig is almost empty:<br><br>rabbitmqctl list_queues memory | awk '{T += $1;}END{print T;}'<br>5703040<br><br>this means that the total memory used for queues is arround 5Mb but RES memory for beam.smp process is 358Mb !!!<br>
<br>And when I start using it, things get worse. beam.smp just jumps from 400m to1.6G (in jumps of 100-200m per second) and well... my VPS provider just decides to kill almost everything on that machine. Right now I'm killing the B process after a bunch of messages, but actually i don't know how to estimate how much memory rabbit will be using per message.<br>
<br><br>I've heard the "new persister" (what a name, eh!) is "much better" with "memory issues":<br><a href="http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-May/007251.html">http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-May/007251.html</a><br>
<br>taken from:<br><a href="http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-February/006397.html">http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-February/006397.html</a><br><br>> =INFO REPORT==== 16-Feb-2010::04:42:05 ===<br>
> Memory limit set to 3196MB.<br><br>I'd love that functionality! when was implemented? or what parameters I need to pass to rabbit?<br><br>Greetings!<br><br clear="all">Nico César<br><br>