[rabbitmq-discuss] More throughput using cluster

[Nikola Savic]

php-amqplib was first solution we used. However, we had to switch to 
AMQP extension because connecting was very slow :( It took minimum 50ms 
for php-amqplib to establish connection to RabbitMQ. Most of this time 
was spent on fread call. AMQP extension also uses a lot of time for 
connection, but much less than php-amqplib (up to 20ms).

We're not using basic_consume, since we must monitor two different 
channels/exchanges (one direct and one fannout). This is because workers 
must get messages (task related) by direct queue, and broadcasts over 
fannout. That's why we used code from one of examples:

/    while (count($ch->callbacks)) {//
//        $read   = array($conn->getSocket()); // add here other sockets 
that you need to attend//
//        $write  = null;//
//        $except = null;//
//        $num_changed_streams = stream_select($read, $write, $except, 
rand(10,15));//
//
//        ....//
//    }/

This is where php-amqplib is much better than AMQP extension :( Since 
AMQP extension doesn't have (or we didn't find example) that can wait on 
stream for predefined number of seconds, and then return control back to 
PHP script, like /stream_select /does.

So, on end, we ended up using AMQP extension for producers and 
php-amqplib for consumers ... crazy isn't it?

On 05/19/2013 04:28 PM, Alvaro Videla wrote:
> Hi,
>
> The php-amqplib from here https://github.com/videlalvaro/php-amqplib 
> can easily publish 4000 msgs in 1.5 seconds and it can consume them in 
> about the same time, using just 1 producer and 1 consumer.
>
> Is worth noting that the consumers in the benchmark code are doing a 
> basic_consume on the queue and not a basic_get. The benchmark code can 
> be easily run if you check out the library form Github and run *make 
> benchmark*.
>
> I think the problem you have on the producing side is the way PHP 
> work, where sharing resources is not so easy due to the script state 
> being cleaned after every request, thus the need of opening one 
> connection per request.
>
> Regarding persisting connections with PHP, while the socket connection 
> to RabbitMQ could be persisted across requests, what do we do about 
> channels and the whole connection state? With PHP this is not so easy 
> to solve.
>
> If you need to achieve max speed from your publishing side I would 
> consider using a REST endpoint as someone suggested already.
>
> On the other hand if your concern regarding publishing speed is to not 
> make the users wait for the request to finish, then keep in mind that 
> php-fpm has a function called fastcgi_finish_request. You can call it 
> at the end of your script and then send the messages to RabbitMQ.
>
> Regards,
>
> Alvaro
>
>
>
> On Sun, May 19, 2013 at 4:11 PM, Nikola Savic <niks at osic.rs 
> <mailto:niks at osic.rs>> wrote:
>
>
>     Consumers are not restarting ... producers are. Every page request
>     is new execution of PHP script, which must open new connection
>     when trying to send first message, and close it when it finishes.
>
>
>     On 05/19/2013 04:08 PM, Michael Klishin wrote:
>>
>>     2013/5/19 Nikola Savic <niks at osic.rs <mailto:niks at osic.rs>>
>>
>>         Yes ... it's the same app which postpones execution of some
>>         actions using MQ.
>>
>>
>>     Then cluster won't solve your problem because the issue is not
>>     node delivery rate.
>>
>>     You need to avoid constant restarts and reconnections for your
>>     consumers or simply
>>     increase their number and/or throughput.
>>     -- 
>>     MK
>>
>>     http://github.com/michaelklishin
>>     http://twitter.com/michaelklishin
>>
>>
>>     _______________________________________________
>>     rabbitmq-discuss mailing list
>>     rabbitmq-discuss at lists.rabbitmq.com  <mailto:rabbitmq-discuss at lists.rabbitmq.com>
>>     https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>     _______________________________________________
>     rabbitmq-discuss mailing list
>     rabbitmq-discuss at lists.rabbitmq.com
>     <mailto:rabbitmq-discuss at lists.rabbitmq.com>
>     https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130519/744970d2/attachment.htm>