<div dir="ltr"><div>Hi,<br><br>I've finished the first production ready iteration v0.1 of lua-resty-rabbitmqstomp library. I've renamed the library after concerns from Brian;<br><br><a href="https://github.com/wingify/lua-resty-rabbitmqstomp">https://github.com/wingify/lua-resty-rabbitmqstomp</a><br>
<br></div>After correct usage of cosocket api, I'm able to publish concurrently without any errors as seen in our lab tests. The throughput rate is still low when receipts (or confirms) are enabled due to STOMP over AMQP overhead so we've decided to go with a different architecture which involves Redis, message aggregation and RabbitMQ for fast and reliable messaging. For this work, we've written a transport agent called agentredrabbit which is already being used in our production pipelines and we are planning to opensource it soon.<br>
<br><div>Regards.<br></div><div><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, May 31, 2013 at 12:40 AM, agentzh <span dir="ltr"><<a href="mailto:agentzh@gmail.com" target="_blank">agentzh@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello!<br>
<div class="im"><br>
On Thu, May 30, 2013 at 11:56 AM, Rohit Yadav wrote:<br>
><br>
> I think this could be a reason, due to RabbitMQ's disk io for persisting<br>
> queue/messages.<br>
> In that case would it be too much performance penalty to sleep for a while<br>
> and retry with a new connection in the request?<br>
<br>
</div>Explicit sleeping may be too much, because the waiting latency is not<br>
under complete control.<br>
<br>
I've been thinking about some kind of automatic request queueing in<br>
ngx_lua's cosocket connection pool so that your tcpsock:connect() call<br>
can just temporarily hang up waiting for other connections to complete<br>
when the backend request concurrency limit is hit. That way, the<br>
waiting latency involved cannot be any longer than necessary and<br>
client traffic peaks cannot overload the backend services.<br>
<br>
Before that happens, the standard ngx_limit_req and ngx_limit_conn<br>
modules can also temporarily block exceeding client connections or<br>
request, thus effectively limiting the backend request concurrency<br>
level.<br>
<div class="im"><br>
> To make the request finish fast I can do a pcall(ngx.eof) and end the<br>
> request and then carry on with publishing message to the RabbitMQ broker.<br>
><br>
<br>
</div>This is a common trick used by the community. Alternatively, you can<br>
also use <a href="http://ngx.timer.at" target="_blank">ngx.timer.at</a>() to do async job processing:<br>
<br>
<a href="http://wiki.nginx.org/HttpLuaModule#ngx.timer.at" target="_blank">http://wiki.nginx.org/HttpLuaModule#ngx.timer.at</a><br>
<br>
But always keep in mind, async processing can be a devil because jobs<br>
can accumulate really fast and eventually exhaust the system resources<br>
if the backend cannot catch up with the frontend. This is also an<br>
effective DoS attack.<br>
<br>
Happy hacking!<br>
<div class="HOEnZb"><div class="h5"><br>
Best regards,<br>
-agentzh<br>
<br>
--<br>
You received this message because you are subscribed to the Google Groups "openresty-en" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:openresty-en%2Bunsubscribe@googlegroups.com">openresty-en+unsubscribe@googlegroups.com</a>.<br>
For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank">https://groups.google.com/groups/opt_out</a>.<br>
<br>
<br>
</div></div></blockquote></div><br></div></div>