[rabbitmq-discuss] FW: One Producer, Many Consumers, Manyphysical nodes

[Jerry Kuch]

Hi, Paul... 




In re (1): exactly Jerry, application specific. I see the queue as containing useful information about the application's progress and its current state. Hence, I didn't want a message to be deleted "merely" because the worker consumer pulled it from the queue. 

Gotcha. The right conceptual way to think of the deliver-ACK-dequeue lifecycle for a message is 
as implementing a "transfer of responsibility" so that things are left, in the case of client failure, 
in a state that will allow life to meaningfully go on. Thus, the RabbitMQ broker secretes away its 
knowledge of a delivered message until the client sends its ACK, thereby saying "I've got this, you 
can stop worrying about it" at which point Rabbit knows it's no longer under any obligation to 
deliver that message in the future and can dispense with whatever record of it that it has. 


As noted earlier, I am (but for some modest "academic" exposure) new to MOM. Using a queue as a persistence store may not be the best design idea. My thinking is that the data is there in the queue, so why incur the overhead of copying it to the lowest layer database store? I envision an external bit of software, e.g., a reporting tool, being able to query the contents of a queue. I suppose it would suffice if the reporting tool could read through the queue sequentially (though the ability to access a single message directly would be nice). 

Although it might seem that there's some initial economy in this mode of thinking, it's probably 
not an approach that will serve your distributed app design super well as you go forward. 
Rabbit's good at accepting and delivering messages, and providing understandable (although 
not always totally obvious without some study or reflection) guarantees of what conditions that 
will happen under. 

If you're wanting to persist data for the ages, or for offline batch processing, or for generating 
interesting reports or analytics, you're best off using a data store suited to feeding whatever 
systems or engines are doing that work. Rabbit makes a fine way to transport such stuff 
between the various systems that need to handle it, e.g. things that write into logs or databases, 
workers processing jobs represented by messages out of a queue, etc. Think of Rabbit as 
being a tool you can use for creating sort of the "peripheral nervous system" of your application, 
ferrying tasks, events, notifications and the like between whatever your system's eyes, hands, 
feet, brains, tentacles, etc. might be. It's likely your various applications requiring long term 
persistence, or richer querying and processing, aren't going to map well on to the message 
transport system itself, and you might find yourself painted into strange corners if you try 
to conflate the intention of these different types of systems too readily. Make sense? 


But maybe I am thinking about this the wrong way. Perhaps the only way to obtain messages from a queue is to be a proper consumer...? 

This is basically the case. Sneak and peek tricks on queues, e.g. by consuming a batch of 
messages, not ACK-ing them, and then rejecting them so that they requeue can quickly 
lead to systems whose behavior isn't that easy to reason about. If you think about Rabbit 
as a way of loosening the coupling between systems that produce and eat data, you'll find 
your cutting with the grain much more of the time and it will probably become easier to 
focus on the parts of your system that do the work, as you let Rabbit gracefully move into 
the background keeping things tied together and running, as producers and consumers may 
be coming and going, starting and crashing, etc. 

Best regards, 
Jerry 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120119/6efb3321/attachment.htm>