[rabbitmq-discuss] Publish won't work without transaction?

[Ben Hood]

Tsuraan,

On Tue, Sep 23, 2008 at 12:13 AM, tsuraan <tsuraan at gmail.com> wrote:
>> So how do you know whether they're enqueued or not?
>
> When the program's done running, I do a passive queue_declare,

Is there a reason why you are passively declaring the queue?

> and the
> message count has increased by ~400 each time the program has run.
> It's actually more like 300 the first time and 500 the second time,
> but it seems like overall, I get about 400 per run.

I still don't see how you are working this out - are you using
rabbit_amqqueue:stat_all() or are you maintaining a count with your
consumers?

> Ok, so there isn't a return code from basic_publish that will tell me
> it was dropped?

basic.publish is an asynchronous command.

If you register a return handler and set the mandatory flag, you will
get a return if the message can't be delivered.

>> That shouldn't take a minute even with a TX per message and logging to disk.
>>
>> I think something may not be quite right with your setup.
>
> I'd be willing to believe that.  When running with one transaction per
> message, my system is almost idle.  python's taking 2-3% of the cpu,
> beam.smp is taking less than that, and my IO is a few KB/s.  Do you
> have any advice for figuring out what's going on?  My system is a
> core2 duo with a pair of raptor10k drives in RAID1 on a 3ware 9650; I
> shouldn't be too starved for cpu, io, or memory with this setup, so
> any advice for where to start would be much appreciated.

Hmmm.....that's better than most pizzaboxes I have to deal with :-)

I don't thing your kit is the issue.

> Is the difference then that my queues are persistent?  I'm using a
> topic exchange with just one persistent, non-autodelete queue.  Does
> that change things?

No, a persistent queue is one that survives a server restart. Whether
or nott a message survives a restart depends on the flag that is sent
with the message header.

Persisting a message obviously has an overhead, but it should be a
reasonably constant factor.

>
>> And maybe this is a stupid question, but why are you enqueuing stuff?
>> Don't you want it to be delivered?
>
> Isn't enqueuing a necessary step before delivery?

The main goal of a messaging system is to deliver messages rather than
queue them up. So Rabbit will only queue if it can't deliver.

In this light, you should be able to see the effect of the immediate
flag - if a message cannot be delivered immediately, bin it.

So you need to have something reading off the queues to avoid the
message being turfed. Or don't set the immediate flag.

In any case, it's not a good idea to enforce queueing - you will build
up an unecessary backlog.

> The system I'm
> working on processes a lot of files, and as files are processed new
> files or database rows are created that have to be processed
> downstream.  Sometimes the programs doing the processing get hung up
> on invalid data, or get flooded with more data than they can process
> in a timely manner, so the workloads get really badly backed up.  I've
> seen programs that have millions of tasks in their job queues more
> than a few times.

If this is the case, you probably don't want to set the immediate
flag, unless you catch it using the return handler - but this is
clunky in itself anyway.

As indicated in a previous thread that Edwin started, there is a
finite limit as to how much Rabbit can queue things up.

You should calibrate your system and don't overfeed the bunny accordingly.

There is a roadmap item to address this - but no ETA.

> We currently have a "message queue" system that is just files written
> to the hard drive; different directories are the job queues for
> different actors, and the files in them are just identifiers for
> database or file-based workloads.  I want to replace that with real
> message queue system for all sorts of reasons, so I'm just
> experimenting with RabbitMQ to see if it does what I need, and to see
> how good the performance and stability are.

Doesn't sound like an unreasonable use case. I can think of different
approaches, but Rabbit may provide you with a simple solution.

HTH,

Ben