[rabbitmq-discuss] Publish won't work without transaction?

Tue Sep 23 20:28:14 BST 2008

> Is there a reason why you are passively declaring the queue?
>
...
> I still don't see how you are working this out - are you using
> rabbit_amqqueue:stat_all() or are you maintaining a count with your
> consumers?

The response from the passive declaration gives how many messages are
in the queue.  That's how I'm checking the number of messages that
exist.  There isn't a consumer yet; I've just been trying to make
message insertion work, or at least understand how it's supposed to
work.  Maybe I'm not making sense here - see below.

>> Ok, so there isn't a return code from basic_publish that will tell me
>> it was dropped?
>
> basic.publish is an asynchronous command.
>
> If you register a return handler and set the mandatory flag, you will
> get a return if the message can't be delivered.

Ok, I'll have to look into this.

> The main goal of a messaging system is to deliver messages rather than
> queue them up. So Rabbit will only queue if it can't deliver.
>
> In this light, you should be able to see the effect of the immediate
> flag - if a message cannot be delivered immediately, bin it.
>
> So you need to have something reading off the queues to avoid the
> message being turfed. Or don't set the immediate flag.

Ok, I'm definitely having terminology issues, and I can't find the
definitions in the AMQP-0.8 spec.  A producer publishes a message to
an exchange.  An exchange routes a message into a queue (or many
queues).  A consumer gets a message from a queue, and once it has
acknowledged that message, then the message is delivered.  Is that
right?  My understanding was totally non-sensical, which probably made
this discussion a bit more confusing for everybody involved than it
had to be...

If all consumers are using basic_get instead of basic_consume, will
immediate delivery always fail?  In other words, if a queue has no
"consumer" on it, will all messages marked for immediate delivery be
dropped?

> In any case, it's not a good idea to enforce queueing - you will build
> up an unecessary backlog.

Yeah, the backlog is exactly what I need to manage though, because my
backlog management isn't the best.  That's essentially the problem I
was hoping that RabbitMQ would solve; apparently it doesn't do that
yet, but it's still much better designed than my way, and I'm guessing
it's probably higher performance.  I have a bit of Erlang experience;
how difficult do you think it would be to implement a custom queue
that allows for ludicrous backlogs, once the upcoming RabbitMQ version
with pluggable queues is done?

> If this is the case, you probably don't want to set the immediate
> flag, unless you catch it using the return handler - but this is
> clunky in itself anyway.
>
> As indicated in a previous thread that Edwin started, there is a
> finite limit as to how much Rabbit can queue things up.
>
> You should calibrate your system and don't overfeed the bunny accordingly.

I can probably hack my way around this, but the tasks my system gets
aren't requested; it just gets tasks as they are generated.  Often, a
system will get a lot of work during the day, and then just process it
all night.  Sometimes, a system will just get too much data, and never
be able to process it all.  Then, the customer calls tech support, but
no data is lost.  The pluggable queues would probably be the right
answer for this, I guess.

> There is a roadmap item to address this - but no ETA.

Is that item the pluggable queues, or an actual implementation of
queues that can grow indefinitely?

> Doesn't sound like an unreasonable use case.

I don't think it sounds unreasonable either, but I've been reading
about file system gotchas lately, and one common fs issue (I sure wish
I could still find my source for this) is that creating and deleting
files forces a metadata sync.  This tends to thrash the drives
excessively, which is a problem we've been having lately.  I'm going
to try implementing our system using rabbit, and have another one
using the fs-based queues, and see if hard drive througput increases.
Even if that doesn't fix that performance issue, I really like the
exchange and queue design of AMQP; it would really let us clean up a
lot of stuff in our system.