[rabbitmq-discuss] RabbitMQ Consistent Hashing Exchange

[John Turner]

Hi Tim,

I've added some comments inline.  I'm very happy to help you with 
requirements here so if a call would help let me know and I'll set it up.

Do you have a timeline for this feature?

Regards

John

On Wednesday, 3 July 2013 21:24:26 UTC+1, hyperthunk wrote:
>
> We've got an open bug to provide a 'message groups' feature like this and 
> are hoping to get it ready for a feature release ASAP. 
>
> Just out of interest, for your use-case, how would you expect failover to 
> work.

 

> Do you want all future messages delivered to another random consumer? 
> (Yes)

 

> What about un-acked messages that were already delivered - we plan on 
> redelivering these (Agree) - and how do you plan on explicitly 
> terminating a group? (This is not a requirement.  Groups should only be 
> terminated by a connection closing either explicitly or due to failure)

 

> Would a special message (or header) given by the producer suit that 
> purpose? (If people want this functionality I would suggest this is the 
> best way.  You might also consider giving the consumer the ability to 
> terminate a message group but we don't have a specific use case for this. 
>  It might be something for future releases.)

 

> What about consumers that don't disconnect but indicate they'd like to 
> stop processing that group of messages? (I referred to this above.  I'd 
> suggest you add this to a future release as it really is not a must have).Do you have consumers that want to recieve more than one group in parallel? 
> (Absolutely) And what about producers that go away - do you close the 
> group when they disconnect or keep it open indefinitely, or use a timeout? (I 
> would keep it open indefinitely because multiple producers may be 
> contributing messages to a single group.  You have some challenges around 
> clean up here though.  It could be reasonable to say that when all messages 
> in a group have been processed that the group closes after a timeout.  When 
> subsequent messages arrive the group is recreated.)
>
> None of those things are very clear to me from the activemq docs, so 
> hearing from a real world user would be nice! :) 
>
> Cheers, 
> Tim 
>
> On 3 Jul 2013, at 11:57, John Turner <john.... at thoughtforge.net<javascript:>> 
> wrote: 
>
> > Hi All, 
> > 
> > I'm very familiar with JMS and have used the GroupID extension to 
> provide processing affinity to a single consumer.  For further information 
> on how this works see this ActiveMQ documentation: 
> http://activemq.apache.org/message-groups.html 
> > 
> > This solves the following requirement: 
> > 
> > * Messages are distributed to a queue with a GroupID header. 
> > * The broker guarantees that messages with a specific GroupID (say 
> '1234') will only be processed by a single consumer. 
> > * If the consumer fails (i.e. the connection is broken or closed), the 
> broker will allow another consumer to process messages with that GroupID 
> > 
> > Using RabbitMQ and the Consistent Hashing Plugin does not not appear to 
> solve the problem because if we are using static (durable) queues, the 
> consumer failover mechanism needs to be implemented outside the server. 
>  Does anyone have any insight into this problem they could share? 
> > 
> > An alternative would be to extend the queue to ensure apply the same 
> consistent hashing mechanism to the actual queue rather than the broker. 
>  Does anyone know if it si possible to extend the queue implementation? 
> > _______________________________________________ 
> > rabbitmq-discuss mailing list 
> > rabbitmq... at lists.rabbitmq.com <javascript:> 
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss 
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130704/bbd8fd20/attachment.htm>