Hi all,<br><div><br></div><div>I'm designing a small distributed system with some queueing involved and have some doubts that Rabbit is the right solution for parts of it. Hoping that some of you can shed some light and let me know if Rabbit can do it all, which would be great, or if there's a better and/or simpler solution I've missed. <span class="Apple-style-span" style="font-family: arial, helvetica, sans-serif; ">I realize that Rabbit is for message passing and is not a work queue, so perhaps I'm trying to shove a square peg into a round hole. If so please tell me that I'm barking up the wrong tree.</span></div>
<meta charset="utf-8"><div><br></div><div>The system is pretty simple and will only have 2 persistent queues, 1 producer, and 1 consumer (initially anyway). Our frontend machine will act on requests from our users and queue up jobs on request. The worker machine will get jobs from that queue, process them, and then place the results in a 2nd queue which the frontend machine will consume, making the results available to our users. So far it's pretty straightforward and RabbitMQ handles this with ease.</div>
<div><br></div><div><font class="Apple-style-span" face="'courier new', monospace">,----------, ,--------,</font></div><div><font class="Apple-style-span" face="'courier new', monospace">| | --- Queue #1 ---> | | Work to be done goes in Queue #1</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">| Frontend | | Worker |</font></div><div><font class="Apple-style-span" face="'courier new', monospace">| | <--- Queue #2 --- | | Completed work goes in Queue #2</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">`----------` `--------`</font></div><div><font class="Apple-style-span" face="'courier new', monospace"><br></font></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif">Now real life has to come along and mess it all up. </font><span class="Apple-style-span" style="font-family: arial, helvetica, sans-serif; ">When a worker is struck by lightning in the simple system above the job it was working on would be lost, which is bad. To me it seems like we need a 3rd "queue" that holds jobs that are currently being processed, and a timeout so that if a job being processed hasn't completed within N minutes it's put back into the first queue so another worker can pick up the job. This queue must also be persistent. The problem is that this 3rd thing isn't really a queue at all. I'd like to remove any specific item from this thing, whether the job was completed or timed out, and thus I need to find and remove arbitrary messages/items.</span></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><div><span class="Apple-style-span" style="font-family: arial, helvetica, sans-serif; "><meta charset="utf-8"><span class="Apple-style-span" style="font-family: arial; "><div>
<span class="Apple-style-span" style="font-family: arial, helvetica, sans-serif; ">I feel like I need to write a bit of code that handles the list of jobs-in-progress and persists it to our main data store (distributed, backed up, etc). If I do all of that when not just put the other 2 queues in our main data store as well? Rabbit still buys us a little for those 2 queues, but not very much. It's easy enough for me to add authenticated HTTP endpoints and distributed, fault-tolerance persistence for a couple of queues.</span></div>
</span></span></div><div><span class="Apple-style-span" style="font-family: arial, helvetica, sans-serif; "><br></span></div><div><meta charset="utf-8"><div style="font-family: arial; "><span class="Apple-style-span" style="font-family: arial, helvetica, sans-serif; ">Is there some other solution that makes this easier, or does Rabbit have something to handle these sorts of cases? If I can't use Rabbit for all of this it may not be worth adding Rabbit to our infrastructure for the 2 very simple queues illustrated above.</span></div>
<div style="font-family: arial; "><span class="Apple-style-span" style="font-family: arial, helvetica, sans-serif; "><br></span></div></div><div><span class="Apple-style-span" style="font-family: arial, helvetica, sans-serif; ">Thanks for reading this far :)</span></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">-s</font></div>