[rabbitmq-discuss] fanout exchange that does not keep messages

[Matthias Radestock]

Brian,

Brian Whitman wrote:
> Matthias -- I'm not sure how I'm seeing them in the queue list when
> there are no consumers. And relatedly perhaps, the broker will crash
> after 20 hours or so when it runs out of memory, which confuses me as
> the queue lengths are always 0.
> 
> Here's my steps using the following code:
> 
> http://notes.variogr.am/post/143623387/broadcasting-your-logs-with-rabbitmq-and-python
> 
> 1) Declare a Producer() and start sending messages 
>  - this does a channel access request, declares the fanout exchange w/
> auto_delete=True, then declares the "logger" queue with exclusive False
> and auto_delete as True, then binds the exchange and the queue
> 
> At this point logger shows up in list_queues with 0 messages. There are
> about 8 machines all writing messages to the queue (using immediate=True
> on basic_publish.) There are no consumers at this point but the queue
> "logger" is in the list.
> 
> 2) Declare a few Consumers() 
>  - this does the same channel access request, declare, bind as the
> Producer() -- same auto_delete and etc except the queue name is now
> "logger-randomUUID" per Conusmer
> 
> At this point logger and any logger-randomUUID show up in the
> list_queues. Each has 0 messages, always. Everything works as it
> should... consumers get messages from the Producers, queue size stays 0. 
> 
> But...
> 
> Memory keeps going
> up -- and eventually the broker will crash. Yesterday it lasted about 20 hours. I booted it again this a.m. and am testing with abnormal message loads (sending millions of messages.) Right
> now memory is at 3.9% and increasing after only running 30m.
> 
> I am using a newer version on the server, yes, and I will do a test on
> the release version to see if that behavior changes. But my guess is I
> am doing something else wrong in my queue setup.

Your setup looks basically ok, except for two things:

1) no consumers ever subscribe to the "logger" queue. That means it
would normally keep accumulating messages. The only reason it doesn't is
because you set immediate=true, but that's generally not the best way to
go about suppressing message storage. Why do you have the logger queue
at all?

2) You are setting immediate=true. I'd advise against that, for reasons
I explained earlier. Setting exclusive or auto_delete on the queues, so
that they disappear when the consumers vanish, should be sufficient.

Neither explains a gradual growth in memory usage. My best guess atm is
that there is a bug in the particular branch of the code you are working
on. So please try the 1.6.0 release. Also, if you still see the problem
then, please post precise instructions on how to reproduce it - a
transcript of a command line session showing how you are invoking the
various producer and consumers scripts would be great.


Regards,

Matthias.