[rabbitmq-discuss] Architectuaral Problem - What exchange type to choose?

Sun Oct 27 11:30:33 GMT 2013

Hi,

we are currently using ActiveMQ with message selectors which are in fact discouraged and having weird results (like blocked consumers) and bad performance. Since I’m about to rewrite parts of the server in Erlang, I’m also evaluating Rabbit as the new MQ system. I’ve read the tutorials and parts of the other docs and trying to figure out how I could solve our problems with Rabbit. Since I couldn’t, I’m writing to you and you could point me in the right direction.

Just a small summary of what needs to be done, later I explain how I plan to realise it using rabbit:

Domain:
We have Devices, lets say D1, D2, D3, D4 etc.. which are grouped in device groups, lets say Group “GA" has D1, D2 and D3 (devices are unique and can only belong to one group):

Group “GA”
- Device “D1"
- Device “D2"
- Device “D3"
Group “GB”
- Device “D4”
(…)

* These devices are embedded devices and are not alway connected to the server. 
* A device may leave a message and/or retrieve pending messages when it is connected. 
* A group owner is also not connected to the server all the time and may connect to retrieve pending messages from a device or leave a message for a device.

Use Cases:
So we have the following use cases in short:
- D->G: Device “D1" connects and leaves some messages M1, M2 and disconnects. 
- The owner of group “GA” connects, retrieves pending messages from “D1” (i.e. M1, M2).
- G->D: The owner of group “GA” leaves a message for device “D1”.
- D->D: Device “D2” connects and leaves a message for “D3”
- D->* : Device “D3” connects and leaves a message broadcasted to all devices of the group (i.e. D1 and D2)

Possible Implementation with Rabbit:
Each device has an in persistent queue, called “<groupname>.<devicename>.to”. 
The group owner has a persistent queue for each device “<groupname>.<devicename>.from”.
(Queue name may actually be exchange names)

- When a device connects and wants to leave a message for the group owner, it creates a producer for queue “G1.D1.from”  and publishes the message.
- When the group owner GA connects and wants to leave a message for D1, it creates a producer for queue “G1.D1.to”  and publishes the message.
- When a device connects and wants to retrieve pending message from the group owner, it creates a consumer for queue “G1.D1.to”  and consumes the messages.
- When the group owner GA connects and wants to retrieve pending messages from D1, it creates a consumer for queue “G1.D1.from”  and consumes the messages.
- When D1 wants to send a message to D3, it creates a producer for “G1.D3.to”
- When D3 wants to broadcast a message, it creates a producer for “G1.*.to”

Now, the first parts ok, the problem may arise when direct message sending and broadcasting is involved.
Intuitivly, I would be looking for an exchange to would support dynamic routing, so that I could use the routing keys with wildcards for a message “G1.*.to” which would route the message to all “to” queues from all devices in group G1. But the bindings and work the other way a round. You must have fixed routing keys for a message and wildcards in bindings.

Is there a way to achieve this with existing rabbit feature and exchange types or do I have to build my own exchange? May with an other architecture, e.g. not using named queues but more sophisticated exchanges etc.

Thanks a lot, especially when you’ve read the whole mail :)

Jan