[rabbitmq-discuss] Headers Exchange Logic

Fri Jul 17 15:55:07 BST 2009

Hi Ben,

> We did look at this briefly, but decided to shelve this feature
> because it is very difficult to implement on the broker side because
> of the multi-dimensional indexes you'd have to maintain.
>
> Having said that, maybe implementing it on the broker side is perhaps
> being a bit too fancy - maybe you can just trade off network bandwidth
> for broker side query complexity.
>
> Having said this though, you'd probably want a way to reject messages
> without having to nuke the channel.

Understood.  Thanks for the clarification.  So currently, custom
header processing such as this would require all applicable consumers
receive a copy of the message with the expectation that 1 consumer
will process the message, while the remaining consumers would simply
drop it on the floor.  Yes, essentially, that would work -- the only
downside is the additional CPU processing required by each consumer
(that evaluated and dropped the message, so N-1 consumers total),
along with the network bandwidth associated with sending the messages
to the (N-1) consumers.

This solution is feasible, as long as N remains relatively small.
There's a point, though, where N becomes large enough to want this
functionality pushed to the AMQP server.

Hi Tony,

> Absolutely -- there's a hugely rich vein of possible exchange behaviours
> waiting to be explored. Streaming SQL? Javascript-based message
> matching? The sky's the limit :)

Okay, I'm glad it's something on your radar.  I realize this type of
support can quickly become extremely complicated, so I understand why
it's not developed just yet.  One thing to keep in mind is that as you
support more complicated evaluations, you'll require the AMQP server
to generate more and more CPU cycles per message.  If you're in an
environment where you don't entirely trust all the consumers, then you
probably want some sort of sanity checking on the evaluation logic
each consumer supplies the server during a bind request.  For example,
if the pseudo code for the bind header were something like:
"while(true) { header > 2; }" then the first evaluation becomes a DoS
attack.  Thus, you'd probably want to set a max evaluation timeout and
if during any message processing, the eval logic takes more than X
amount of time --- *keep* the message in the queue, but *destroy* the
queue/binding of the offending consumer, so that the consumer is
penalized but the message can still be (potentially) reliably
delivered.

I'm sure there's a lot more complexity to deal with than just this
example, but I hope it helps.

Regards,
-- Darien