<div dir="ltr">Thank you for the support Michael.<div>So my guess were right..The problem is that in my use case I should start receiving messages until I receive a special "end-message" (true/false attribute on a json object).</div>
<div>When I receive this end-message I should start a map/reduce job on received data (that gets stored in HBase in this first step).</div><div>Obviously I need a way to ensure that no worker is still working on the data, so the consumer receiving the end-message should be able to wait for all consumer to finish their work.</div>
<div>In my current implementation I use a single queue and a single multi-threaded consumer (via Spring-AMQP) that allow me to achieve this goal.</div><div>But, I'm not sure what's the best way of scaling..is there some RabbitMQ mechanism that I can exploit?</div>
<div>Should I change my design?</div><div>Maybe I should create a dedicated exchange per source and add more queues as the load grows..but here I still got problems when I receive this end message..</div><div>how can I ask Consumers if they finished their work? How add more queues when a certain load is achieved? How remove them when they are no more requested?</div>
<div><br></div><div>Best,</div><div>Flavio</div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 24, 2013 at 10:50 AM, Michael Klishin <span dir="ltr"><<a href="mailto:mklishin@gopivotal.com" target="_blank">mklishin@gopivotal.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Flavio Pompermaier:<br>
<br>
> From what I understood queues cannot be splitted among different machines, a single queue resided only on a single node..am I wrong?<br>
<br>
Queue contents are not sharded but can be mirrored.<br>
<br>
> What if I discover that my machine is not able to keep up with message rate?<br>
<br>
Use multiple queues. A single queue certainly will have scalability limitations.<br>
<br>
> Maybe I should start mirroring the queue on multiple servers so I can multiply the number of consumers?<br>
<br>
Mirroring won't help. Increasing the # of consumers and/or queues likely will. Mirroring is an HA<br>
feature.<br>
<br>
> Is this approach scalable or does the synchronization overhead kill the performance?<br>
<br>
Mirroring involves moving more data between cluster nodes so yes, it does affect cluster<br>
throughput.<br>
--<br>
MK<br><br></blockquote></div><div dir="ltr"><p></p><p></p><p></p><p></p></div>
</div></div></div></div>