[rabbitmq-discuss] Best practice for 1 server, several agents, several kind of messages

[misterdev]

Hello,

First of all, english is not my native language, so I may (will) make some 
mistakes. But I hope everybody will understand me.
I am new to this messaging thing so I am looking for any good advice you 
could give me.

I am going to present you a scenario, and I would really appreciate if you 
could help me, by giving me ideas to implement it. Specifically if there is 
common design pattern I should use :

I have a server application with autonomous agents.
Each agent is on a specific server, and the number of agent can quickly 
increase (today, I have about 100 agents, in a few months, I could have 
1000 if everything goes as planned).
In this scenario, the number of messages per second, for each agent, is 
quite limited : juste a few messages / second / agent.

The dialog between the server application and an agent contains different 
kind of messages that I began to represent with the "topic exchange" kind 
of thing :

// Messages received for a watchdog installed on the agent1 :
agent1.watchdog.critical 
agent1.watchdog.warning

// Messages containing commands that sould be executed my the agent
agent1.cmd.immediate
agent1.cmd.delay

// Messages organizing file transfer between two components of my 
architecture
agent1.fileTransfer.historyFile1.send
agent1.fileTransfer.historyFile1.loaded
agent1.fileTransfer.historyFile1.failure
agent1.fileTransfer.historyFile2.send
agent1.fileTransfer.historyFile2.loaded
agent1.fileTransfer.historyFile2.failure

With this pattern, I will have about 10 to 20 queues for each agent (20 to 
40 if I have a queue for agent to server dialog and another one for server 
to agent dialog).

My initial questions are :
- Will there be a problem with this number of queues in my architecture ? 
(100 agent --> more than 2000 queues) and what should be the size of my 
server hosting the broker ?
- Is there a pattern that could help me ?
- Is it better to use messages headers to identify each message, instead of 
topic exchange ?
- Is it better to differentiate server to agent queue and agent to server 
queue, or is it ok if both read and write in the same queue ?

Note : Because my history files, that have to be transfered between the 
agent and the server, can size several Mo, I don't intend to send them via 
amqp protocol (except if there is no significant performance decrease by 
doing that ?)

To be honest, I read this article :
http://blog.springsource.com/2011/04/01/routing-topologies-for-performance-and-scalability-with-rabbitmq/

And i decided to inspire myself from the "Exchange-to-Exchange Scalable Use 
Case". But I use the perl library Net::RabbitMQ, which does not implement 
exchange to exchange bindings, so I have to modifiy it a little :

So I came with this architecture (see Image below)

I explain a little :
- the server application don't know when an agent will be "awake" and will 
start to talk. So i was thinking of using some kind of authentication thing 
:
    - Each agent, at first start, will begin to talk in a guest exchange, 
declaring and authenticating itself.
    - When the server application authenticates the agent, it will 
automatically bind a new queue to the "client exchange authenticated". 
    - When the agent receive a confirmation of the authentication, it will 
bind about 10 queues to the "server to client authenticated exchange" (with 
the topic exchange described before)
    
Regarding this scenario, what advice could you give me ?

Thanks in advance for any information / advice.

Kilian

The architecture : 

<https://lh4.googleusercontent.com/-4fdI32Uo8cQ/T7IdEM_FehI/AAAAAAAAAHo/6l74h0meAEc/s1600/RabbitMQ+Implementation.jpg>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120515/e5441fa4/attachment.htm>