[rabbitmq-discuss] [help] [beginner] server stops sending messages; publish (in transaction) hangs on commit

[Simon MacMullen]

On 23/02/12 20:00, Alistair Bayley wrote:
> Just ran this:
>
> $ sudo rabbitmqctl eval
> 'lists:sublist(lists:reverse(lists:sort([{process_info(Pid,memory),
> Pid, process_info(Pid)} || Pid<- processes()])), 10).'
>
> (wait for a bit - about 30s or so...)
>
> Crash dump was written to: erl_crash.dump
> eheap_alloc: Cannot allocate 153052320 bytes of memory (of type "old_heap").
> Aborted
> $
>
> hmmm....

Hmm indeed. If it ran out of memory doing that, then that implies you 
might have a ton of processes hanging around for some reason. Either 
that or you were very close to the edge anyway.

rabbitmqctl eval 'length(processes()).' will tell you how many processes 
are in use in total.

So you may want to try the second rabbitmqctl eval command above, and 
then the first one with more memory available to the server.

>> Having said that, 100Mb for an idle server does not seem totally
>> unreasonable - RabbitMQ is not optimised for small memory use when idle.
>
> OK. Can I have some guidance as to how much memory we should allocate?

This is a real how-long-is-a-piece-of-string question I'm afraid. 512M 
would already be a significant step towards the comfort zone though.

> Also, I see from the tcpdump capture that the server does not send
> back a response to the Tx.Commit, which presumably makes it difficult
> for the client to detect that the message has not been accepted (other
> than timeout). Indeed, the basic_publish returns nothing immediately,
> although that could be a feature of the amqplib library. Is there an
> option to get the server to return something that says that it is not
> accepting messages?

In ancient history we used to do this; there's a method in AMQP 0-9-1 
called channel.flow which the server should be able to use to tell a 
client to shut up. In practice it was unreliable (client may not support 
it, or may take too long to take notice of it).

We could just send channel.flows informationally but again this would 
require client support.

I am currently wondering about modifying our clients so they have a 
non-blocking publish (see the thread "Ensuring low latency for 
publishers"). This should not be too hard to do. However, this might not 
help when using transactions. OTOH blocking in such a way that we don't 
send back a tx.commit is unhelpful; I'll see how easy it would be to 
change this.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware