[rabbitmq-discuss] Publish won't work without transaction?

Tue Sep 23 00:13:24 BST 2008

> So how do you know whether they're enqueued or not?

When the program's done running, I do a passive queue_declare, and the
message count has increased by ~400 each time the program has run.
It's actually more like 300 the first time and 500 the second time,
but it seems like overall, I get about 400 per run.

>> Is there a way that I can tell whether the message I sent was dropped
>> by the server?
>
> You can run rabbit_amqqueue:stat_all() in the shell.

Ok, so there isn't a return code from basic_publish that will tell me
it was dropped?

> That shouldn't take a minute even with a TX per message and logging to disk.
>
> I think something may not be quite right with your setup.

I'd be willing to believe that.  When running with one transaction per
message, my system is almost idle.  python's taking 2-3% of the cpu,
beam.smp is taking less than that, and my IO is a few KB/s.  Do you
have any advice for figuring out what's going on?  My system is a
core2 duo with a pair of raptor10k drives in RAID1 on a 3ware 9650; I
shouldn't be too starved for cpu, io, or memory with this setup, so
any advice for where to start would be much appreciated.

> Either with a transaction or by setting up a return handler (and
> setting the mandatory flag).
>
> BTW, when I run this code:
>
> #!/usr/bin/python2.5
>
> import sys
> from time import time
> import amqplib.client_0_8 as A
>
> q = "q-%d" % int(time())
>
> conn = A.Connection("127.0.0.1", "guest", "guest")
> chan = conn.channel()
>
> chan.queue_declare(queue=q, auto_delete=True, durable=False)
>
> for i in range(1000):
>    print i
>    m = A.Message(str(i), delivery_mode=1)
>    print chan.basic_publish(m, "", q, mandatory=True, immediate=False)
>
> and then in the shell
>
> rabbit at xlr8)3> rabbit_amqqueue:stat_all().
> [{ok,{resource,<<"/">>,queue,<<"q-1222121318">>},1000,0}]
>
> which would indicate that 1000 messages got enqueued.

Is the difference then that my queues are persistent?  I'm using a
topic exchange with just one persistent, non-autodelete queue.  Does
that change things?

> And maybe this is a stupid question, but why are you enqueuing stuff?
> Don't you want it to be delivered?

Isn't enqueuing a necessary step before delivery?  The system I'm
working on processes a lot of files, and as files are processed new
files or database rows are created that have to be processed
downstream.  Sometimes the programs doing the processing get hung up
on invalid data, or get flooded with more data than they can process
in a timely manner, so the workloads get really badly backed up.  I've
seen programs that have millions of tasks in their job queues more
than a few times.

We currently have a "message queue" system that is just files written
to the hard drive; different directories are the job queues for
different actors, and the files in them are just identifiers for
database or file-based workloads.  I want to replace that with real
message queue system for all sorts of reasons, so I'm just
experimenting with RabbitMQ to see if it does what I need, and to see
how good the performance and stability are.