[rabbitmq-discuss] Publish won't work without transaction?

[Edwin Fine]

Tsuraan,

I am using durable, non-auto-deleting queues and exchanges with persistent
messages. I'm using the Erlang client in network mode, and not using AMQP
transactions. Every message that I basic.publish makes it to the queue, even
if there is no consumer on the other side draining the queue. No messages
are dropped. I have similar hardware to yours except my disks aren't as fast
:)

Here are my exchange parameters:
:
ExchangeDeclare = #'exchange.declare'{ticket = Ticket, exchange = Exchange,
type = <<"direct">>,
        passive = false, durable = true, auto_delete = false, internal =
false,
        nowait = false, arguments = []}

Here are the queue declare parameters:

    QueueDeclare = #'queue.declare'{
        ticket = Ticket,
        queue = Q,
        passive = false,        % This is a real creation request, not a
status check
        durable = true,         % Messages must be persistent and survive
server restart
        exclusive = false,
        auto_delete = false,    % Don't auto-delete the queue when released
by channel
        nowait = false,         % Wait for a queue.declare_ok response from
the server
        arguments = []},

And the queue bind:

#'queue.bind'{ticket = Ticket, queue = Q1, exchange = X,
                              routing_key = Q1, nowait = false, arguments =
[]}

Here is the basic.publish setup:

#publish{
        routing_key = RoutingKey,
        queue = Q,
        exchange = X,
        bind_key = _BindKey,
        payload = Payload,
        mandatory = false,  % If true, server sends Return if msg
unroutable, else silently drops msg
        immediate = false   % If true, server sends Return if msg cannot be
delivered immediately, else queues msg
    }

On a different topic, if you are expecting to queue millions of persistent
messages to replace a filesystem-based queuing mechanism, you need to be
aware that at present, RabbitMQ keeps all undelivered messages in memory. If
they are persistent, then they are also saved on disk, but my understanding
is that every message is also kept in memory. There is therefore a memory
constraint and you can't just happily enqueue messages without draining the
queues until the disk is full, without risking memory exhaustion and
resultant Erlang node crash. I understand there are plans to implement an
"overflow to disk" mechanism, but the ETA is unknown to me. Ben can say more
about this.

Hope this helps.

On Mon, Sep 22, 2008 at 7:13 PM, tsuraan <tsuraan at gmail.com> wrote:

> > So how do you know whether they're enqueued or not?
>
> When the program's done running, I do a passive queue_declare, and the
> message count has increased by ~400 each time the program has run.
> It's actually more like 300 the first time and 500 the second time,
> but it seems like overall, I get about 400 per run.
>
> >> Is there a way that I can tell whether the message I sent was dropped
> >> by the server?
> >
> > You can run rabbit_amqqueue:stat_all() in the shell.
>
> Ok, so there isn't a return code from basic_publish that will tell me
> it was dropped?
>
> > That shouldn't take a minute even with a TX per message and logging to
> disk.
> >
> > I think something may not be quite right with your setup.
>
> I'd be willing to believe that.  When running with one transaction per
> message, my system is almost idle.  python's taking 2-3% of the cpu,
> beam.smp is taking less than that, and my IO is a few KB/s.  Do you
> have any advice for figuring out what's going on?  My system is a
> core2 duo with a pair of raptor10k drives in RAID1 on a 3ware 9650; I
> shouldn't be too starved for cpu, io, or memory with this setup, so
> any advice for where to start would be much appreciated.
>
> > Either with a transaction or by setting up a return handler (and
> > setting the mandatory flag).
> >
> > BTW, when I run this code:
> >
> > #!/usr/bin/python2.5
> >
> > import sys
> > from time import time
> > import amqplib.client_0_8 as A
> >
> > q = "q-%d" % int(time())
> >
> > conn = A.Connection("127.0.0.1", "guest", "guest")
> > chan = conn.channel()
> >
> > chan.queue_declare(queue=q, auto_delete=True, durable=False)
> >
> > for i in range(1000):
> >    print i
> >    m = A.Message(str(i), delivery_mode=1)
> >    print chan.basic_publish(m, "", q, mandatory=True, immediate=False)
> >
> > and then in the shell
> >
> > rabbit at xlr8)3> rabbit_amqqueue:stat_all().
> > [{ok,{resource,<<"/">>,queue,<<"q-1222121318">>},1000,0}]
> >
> > which would indicate that 1000 messages got enqueued.
>
> Is the difference then that my queues are persistent?  I'm using a
> topic exchange with just one persistent, non-autodelete queue.  Does
> that change things?
>
> > And maybe this is a stupid question, but why are you enqueuing stuff?
> > Don't you want it to be delivered?
>
> Isn't enqueuing a necessary step before delivery?  The system I'm
> working on processes a lot of files, and as files are processed new
> files or database rows are created that have to be processed
> downstream.  Sometimes the programs doing the processing get hung up
> on invalid data, or get flooded with more data than they can process
> in a timely manner, so the workloads get really badly backed up.  I've
> seen programs that have millions of tasks in their job queues more
> than a few times.
>
> We currently have a "message queue" system that is just files written
> to the hard drive; different directories are the job queues for
> different actors, and the files in them are just identifiers for
> database or file-based workloads.  I want to replace that with real
> message queue system for all sorts of reasons, so I'm just
> experimenting with RabbitMQ to see if it does what I need, and to see
> how good the performance and stability are.
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20080922/a3523210/attachment.htm