[rabbitmq-discuss] Queuing for an application that does a fair amount of synchronous processing

[vishnu]

We're also looking at about 3000 messages/second (high side), out of which
we need 1000 of these to be persistent (since they represent payments that
must happen). However all of these are currently meant to be synchronous,
meaning we have someone sitting on a browser waiting for us to deal with
this message and send a response back. So in the short term the only
advantage we're hoping to gain is in the case of the persistent queues. In
this case we're relying on the guarantee that the message will be dealt with
even if there is some temporary glitch, and our web based client gets a
timeout message.

The way we plan to implement these synchronous messages is, similar to the
RPC model, to create a temporary queue when we send out a request and
subscribe to that queue with a timeout for a response, deleting it after
we're done.

now, if the webserver thread breaks out of the wait before the message
returns, my understanding is that the temporary queue gets deleted and the
response from the backend server is sent to the exchange where it dies.
(which is fine)

if the webserver thread breaks out of the wait after the message returns,
the queue gets deleted anyway and the message should vanish.

If the webserver thread dies for some reason, the queue gets auto deleted
and so the  backend server again sends the response to the exchange to a non
existent queue.

However, the problem comes in when the webserver thread dies just as the
response returns on the temporary queue. When this race condition occurs,
the queue should remain on the rabbitmq-server forever. Is this right, or is
there something I'm missing?

On Fri, Jan 1, 2010 at 1:06 AM, vishnu <pathsny at gmail.com> wrote:

> fair enough, let me come back with some numbers :), I just wanted to first
> make sure I'm not doing something brain dead :).
>
>
> On Thu, Dec 31, 2009 at 8:51 PM, Chuck Remes <cremes.devlist at mac.com>wrote:
>
>>
>> On Dec 31, 2009, at 9:15 AM, vishnu wrote:
>>
>> > Hi
>> >    We're looking at building an application that does a fair amount
>> > of synchronous processing. However we are investigating using a
>> > queuing system as a messaging bus because some of these processes
>> > are meant to be reliable. Using a queue helps us guarantee that
>> > messages are dealt with and avoids us having to right background
>> > processes to clean up after us.
>> >
>> > There are, however, a fair number of operations that are not
>> > required to be reliable and we are thinking of implementing these
>> > also through the queue but use non persistent queues. Now the
>> > concern we have is the kinds of latencies we might see. So I'd like
>> > to get general feedback on using this approach.
>>
>> Give us some idea of the data rates you are expecting. I regularly
>> publish 3000+ messages/second using ruby which isn't exactly known for
>> its performance. :)  Latency is a millisecond or so for topic
>> exchanges, and under 1 millisecond for direct and fanout exchanges.
>> (Exchange type matters due to matching overhead.)
>>
>> However, since you say that all processing is synchronous, the length
>> of time to complete that process will be your gating factor.
>>
>> cr
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100106/23afa1ec/attachment.htm