<div dir="ltr">Then you could add a mutex in addition to rate limiting and prefetch twiddling to insure that batch messages are not processed as long as high priority messages are available for processing.<div><br></div><div>
Good luck,</div><div><br></div><div>Michael</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Dec 2, 2013 at 5:25 AM, vadlamani maitreya <span dir="ltr"><<a href="mailto:maitreya_vvm@yahoo.co.in" target="_blank">maitreya_vvm@yahoo.co.in</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:12pt;font-family:times new roman,new york,times,serif">Hi Michael,<br><br>Thankyou for the response. The application we built infact supports only two priorities, ondemand and batch. The problem here is once the ondemand message is in queue, it must be processed irrespective of any number of batch messages. That is the reason for this weird logic we had to employ. For this usecase, I dont think the rate limiting and prefetch count variations would actual work. But I will try to play with the asynch connections for sure.<br>
<br>regards,<br>Maitreya<div><div class="h5"><br><br><div><span><br></span></div><div style="display:block"> <br> <br> <div style="font-family:times new roman,new york,times,serif;font-size:12pt"> <div style="font-family:times new roman,new york,times,serif;font-size:12pt">
<div dir="ltr"> <font face="Arial"> On Tuesday, 26 November 2013
8:32 PM, "Laing, Michael" <<a href="mailto:michael.laing@nytimes.com" target="_blank">michael.laing@nytimes.com</a>> wrote:<br> </font> </div> <div><div><div><div dir="ltr">You say you want a scalable solution. Therefore with pika you should use a high-performance asynch connector, i.e. tornado in pika 9.13. So start there and follow the examples of using asynch connections - forget the blocking connection.<div>
<br clear="none"></div><div>Then you could try this:<br clear="none"><div><br clear="none"></div><div>To do a simple priority queue I set up 2 queues and bind to both. Usually I have one queue that gets big bursts of messages and another, the priority one that only gets a few at a time. In this case I just set prefetch_count to 1 for both queues and let them interleave. I use acks always.<br clear="none">
</div><div><br clear="none"></div><div>To give more 'preference' to the priority queue in the above case you can increase its prefetch_count, increasing the likelihood that a priority message will be ready when the event loop rolls around.</div>
<div><br clear="none"></div><div>For even more preference you can 'rate limit' the non-priority queue. I do this by registering a short timeout after processing each non-priority message which does the ack in its callback.</div>
<div><br clear="none"></div><div>There are much more complicated ways to do this. I prefer simple.</div><div><br clear="none"></div><div>Good luck,</div><div><br clear="none"></div><div>Michael</div></div><div><br clear="none">
</div><div>P.S. You can subscribe to a queue that is not currently bound to anything. Once subscribed, you can bind the queue one or more times and catch each bind_ok callback to know you are consuming. I'm not sure this is actually useful in your case.</div>
</div><div><br clear="none"><br clear="none"><div>On Tue, Nov 26, 2013 at 1:25 AM, vadlamani maitreya <span dir="ltr"><<a rel="nofollow" shape="rect" href="mailto:maitreya_vvm@yahoo.co.in" target="_blank">maitreya_vvm@yahoo.co.in</a>></span> wrote:<br clear="none">
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div style="font-size:12pt;font-family:times new roman,new york,times,serif">Hello Everyone,<br clear="none"><br clear="none">I have a small problem with RMQ and pika client and would like to ask guyz here for some help. The situation is as below,<br clear="none">
<br clear="none">I am using RabbitMQ with the pika clients in a scalable solution. The solution requires messaging with two priorities. So every worker (through pika client) gets messages from the HighPriority Queue first and then goes to the Low Priority queue if there are no messages left in the HP Queue. I decide to scale the number of workers according to the number of ready messages and the number of consumers attached to the queue. So the following is the workflow i follow to achieve the above.<br clear="none">
<br clear="none">1) Register using basic_consume on the queue.<br clear="none">2) Do basic_get from HP Queue<br clear="none">3)If no messages in HP Queue do a basic_get from LP Queue.<br clear="none"><br clear="none">I donot execute start_consuming() so
the basic_consume is just for the number of consumers count in the RMQ server.<br clear="none"><br clear="none">This logic works perfectly fine with the standard Java Client. But when i try to do the same with the Pika Client, the moment the basic_consume is called the data events are processed and the callback function is executed on the incoming message, eventhough the start_consuming() function is not called.<br clear="none">
<br clear="none">So, a couple of questions here,<br clear="none"><br clear="none">1. Is there a way to register as a consumer to a queue without actually starting to consume from the queue (even a hack into the code is fine)<br clear="none">
2. Is it bug that basic_consume() starts consuming without actually calling start_consuming() (the actual blocking function), and should i raise an issue in github regarding this ?<br clear="none">
<br clear="none">Last but not the least, the version of the rabbitMQ server is 3.1.2 and the pika client, i have tried both 0.9.6 and 0.9.13 with the above scenario.<br clear="none"><br clear="none">Thankyou
everyone,<br clear="none">Maitreya<br clear="none"><br clear="none"><br clear="none"><br clear="none"><div><br clear="none"></div></div></div></div><br clear="none">_______________________________________________<br clear="none">
rabbitmq-discuss mailing list<br clear="none">
<a rel="nofollow" shape="rect" href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br clear="none">
<a rel="nofollow" shape="rect" href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br clear="none">
<br clear="none"></blockquote></div><br clear="none"></div></div></div><br><br></div> </div> </div> </div> </div></div></div></div></blockquote></div><br></div>