<div dir="ltr">Tsuraan,<br><br>I am using durable, non-auto-deleting queues and exchanges with persistent messages. I'm using the Erlang client in network mode, and not using AMQP transactions. Every message that I basic.publish makes it to the queue, even if there is no consumer on the other side draining the queue. No messages are dropped. I have similar hardware to yours except my disks aren't as fast :)<br>
<br>Here are my exchange parameters:<br>:<br>ExchangeDeclare = #'exchange.declare'{ticket = Ticket, exchange = Exchange, type = <<"direct">>,<br> passive = false, durable = true, auto_delete = false, internal = false,<br>
nowait = false, arguments = []}<br><br>Here are the queue declare parameters:<br><br> QueueDeclare = #'queue.declare'{<br> ticket = Ticket,<br> queue = Q,<br> passive = false, % This is a real creation request, not a status check<br>
durable = true, % Messages must be persistent and survive server restart<br> exclusive = false,<br> auto_delete = false, % Don't auto-delete the queue when released by channel<br> nowait = false, % Wait for a queue.declare_ok response from the server<br>
arguments = []},<br><br>And the queue bind:<br><br>#'queue.bind'{ticket = Ticket, queue = Q1, exchange = X,<br> routing_key = Q1, nowait = false, arguments = []}<br><br>Here is the basic.publish setup:<br>
<br>#publish{<br> routing_key = RoutingKey,<br> queue = Q,<br> exchange = X,<br> bind_key = _BindKey, <br> payload = Payload,<br> mandatory = false, % If true, server sends Return if msg unroutable, else silently drops msg<br>
immediate = false % If true, server sends Return if msg cannot be delivered immediately, else queues msg<br> }<br><br>On a different topic, if you are expecting to queue millions of persistent messages to replace a filesystem-based queuing mechanism, you need to be aware that at present, RabbitMQ keeps all undelivered messages in memory. If they are persistent, then they are also saved on disk, but my understanding is that every message is also kept in memory. There is therefore a memory constraint and you can't just happily enqueue messages without draining the queues until the disk is full, without risking memory exhaustion and resultant Erlang node crash. I understand there are plans to implement an "overflow to disk" mechanism, but the ETA is unknown to me. Ben can say more about this.<br>
<br>Hope this helps.<br><br><div class="gmail_quote">On Mon, Sep 22, 2008 at 7:13 PM, tsuraan <span dir="ltr"><<a href="mailto:tsuraan@gmail.com">tsuraan@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="Ih2E3d">> So how do you know whether they're enqueued or not?<br>
<br>
</div>When the program's done running, I do a passive queue_declare, and the<br>
message count has increased by ~400 each time the program has run.<br>
It's actually more like 300 the first time and 500 the second time,<br>
but it seems like overall, I get about 400 per run.<br>
<div class="Ih2E3d"><br>
>> Is there a way that I can tell whether the message I sent was dropped<br>
>> by the server?<br>
><br>
> You can run rabbit_amqqueue:stat_all() in the shell.<br>
<br>
</div>Ok, so there isn't a return code from basic_publish that will tell me<br>
it was dropped?<br>
<div class="Ih2E3d"><br>
> That shouldn't take a minute even with a TX per message and logging to disk.<br>
><br>
> I think something may not be quite right with your setup.<br>
<br>
</div>I'd be willing to believe that. When running with one transaction per<br>
message, my system is almost idle. python's taking 2-3% of the cpu,<br>
beam.smp is taking less than that, and my IO is a few KB/s. Do you<br>
have any advice for figuring out what's going on? My system is a<br>
core2 duo with a pair of raptor10k drives in RAID1 on a 3ware 9650; I<br>
shouldn't be too starved for cpu, io, or memory with this setup, so<br>
any advice for where to start would be much appreciated.<br>
<div class="Ih2E3d"><br>
> Either with a transaction or by setting up a return handler (and<br>
> setting the mandatory flag).<br>
><br>
> BTW, when I run this code:<br>
><br>
> #!/usr/bin/python2.5<br>
><br>
> import sys<br>
> from time import time<br>
> import amqplib.client_0_8 as A<br>
><br>
> q = "q-%d" % int(time())<br>
><br>
> conn = A.Connection("<a href="http://127.0.0.1" target="_blank">127.0.0.1</a>", "guest", "guest")<br>
> chan = conn.channel()<br>
><br>
> chan.queue_declare(queue=q, auto_delete=True, durable=False)<br>
><br>
> for i in range(1000):<br>
> print i<br>
> m = A.Message(str(i), delivery_mode=1)<br>
> print chan.basic_publish(m, "", q, mandatory=True, immediate=False)<br>
><br>
> and then in the shell<br>
><br>
> rabbit@xlr8)3> rabbit_amqqueue:stat_all().<br>
> [{ok,{resource,<<"/">>,queue,<<"q-1222121318">>},1000,0}]<br>
><br>
> which would indicate that 1000 messages got enqueued.<br>
<br>
</div>Is the difference then that my queues are persistent? I'm using a<br>
topic exchange with just one persistent, non-autodelete queue. Does<br>
that change things?<br>
<div class="Ih2E3d"><br>
> And maybe this is a stupid question, but why are you enqueuing stuff?<br>
> Don't you want it to be delivered?<br>
<br>
</div>Isn't enqueuing a necessary step before delivery? The system I'm<br>
working on processes a lot of files, and as files are processed new<br>
files or database rows are created that have to be processed<br>
downstream. Sometimes the programs doing the processing get hung up<br>
on invalid data, or get flooded with more data than they can process<br>
in a timely manner, so the workloads get really badly backed up. I've<br>
seen programs that have millions of tasks in their job queues more<br>
than a few times.<br>
<br>
We currently have a "message queue" system that is just files written<br>
to the hard drive; different directories are the job queues for<br>
different actors, and the files in them are just identifiers for<br>
database or file-based workloads. I want to replace that with real<br>
message queue system for all sorts of reasons, so I'm just<br>
experimenting with RabbitMQ to see if it does what I need, and to see<br>
how good the performance and stability are.<br>
<div><div></div><div class="Wj3C7c"><br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br>
</div></div></blockquote></div><br></div>