[rabbitmq-discuss] rabbit MQ high-through put stock quotes

[smittycb10]

This is a follow up to my previous message with some more details about how I
envision using Rabbit MQ.

Currently I have a system where a server takes in all messages from our data
provider and then streams the messages via UDP to a consumer application.
This set up can handle the message load but is proving to be difficult and
costly to scale. One producer to one Consuming server which then filters
data to desktop clients.

My idea for a future system is to have the same server simply push the
messages to a rabbit MQ Broker or Brokers and then have all clients simply
listen to whatever queues they need to get there jobs done. I need to design
for 100,000 (one hundred thousand) messages to made available for consuming
per second. Most consumers will only consume a fraction of these messages,
although some of our server pieces may need to subscribe to the whole
universe of incoming messages. Messages can and will be batched on the
publishing side.

So the use case would be, one publisher pushing messages to the "Broker
Cloud" many consumers some needing to listen to all messages others would
only be interested in a particular Stock symbol at a time. 

I will experiment with some fanout experiments next, any advice or direction
you have would be greatly appreciated. In a fanout pattern if say a quote
comes in for IBM how do I distribute only to people interested in IBM?

smittycb10 wrote:
> 
> Hello,Attached is a very simple version of my original experiment. I have
> been playing around with the messages per-second and seem to top out at
> about 5000 MS per second before the consumer appears to back up. Is this
> normal performance? (one broker running on windows with out of the box
> settings).
> Now what I really want to do is develop a pub sub system for listening to
> stock quotes that can handle 1000,000 messages per-second. I have a 8 core
> windows server 2003 at my disposal, do you think this is possible?
> If so how many brokers do you think I should run?
> What would be the optimal way to arrangement for my exchanges and queues
> to
> maximize use of all cores on the broker machine.
> How can I use direct queues in a publish subscribe system, I ask because
> you
> mentioned that direct queues are more efficient.
> 
> Thanks
> Mark
> 
> -- 
> Mark Smith
> 
> please send all correspondences to this
> new email address
> 
> 
> 
> -- 
> Mark Smith
> 
> please send all correspondences to this
> new email address
> 
>  
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> 
> 

-- 
View this message in context: http://www.nabble.com/rabbit-MQ-high-through-put-stock-quotes-tp23860396p23876536.html
Sent from the RabbitMQ mailing list archive at Nabble.com.