[rabbitmq-discuss] Deploying RabitMQ for my Application - Questions and Doubts

Wed Dec 30 11:02:12 GMT 2009

Karthik

Yes that is feasible.  If you were simply talking about a 'chat'
system then I would suggest XMPP would be great for your case.  But,
the mixture of pubsub, fanout and queueing ("pushing .. as and when
the need arises") suggests that you should consider AMQP.

Re use cases:  I cannot talk about *everyone* doing exactly the same
as you, since many of our customers don't like to (or cannot) discuss
their cases in public.   However, here is a reasonably well known
'multi-location multi-datacenter multi-cloud' use case that is
similar:

http://www.oceanobservatories.org/spaces/display/CIDev/Messaging+Service+Specification+V1
http://io9.com/5410036/fleets-of-autonomous-robots-coming-to-an-ocean-near-you
http://www.calit2.net/newsroom/article.php?id=1505

Here is a case that has several data centers, similar to what you are
asking about: http://qconsf.com/sf2009/presentation/Open+Source+at+Unibet.com+-+10x+Scalability+at+Half+the+Cost

Here are some more 'cloud' use cases:
http://delicious.com/alexisrichardson/rabbitmq+cloud+usecase

 More below...

On Wed, Dec 30, 2009 at 10:39 AM, Karthik <coolkarthik88 at gmail.com> wrote:
> Hi Alex,
> Thanks for writing in. Here is the scenario.. I need to be running a single
> cluster of Servers housed in a single Data Center. The problem I am trying
> to solve involves pushing out information to clients spread across the globe
> asynchronously aka push the data to them as and when the need arises.

OK.  I assume that 'as and when the need arises' means that
connections will be somewhat unpredictable, and some data will need to
be queued for offline consumers?  Will it also need to be stored?

> The
> data to be pushed in will be pumped by a Web Based front end..

Would you want to use one of the RabbitMQ web clients for this?

> The communication will primarily be Private communication, that is server
> directly talks to the client.. In my problem none of the clients can talk to
> each other.. Information can only be exchanged between the client and server
> in both the directions. PubSub or fanout communications will be involved
> too, but most communication will be private (connections need to be
> encrypted).

OK.

> To scale my system, I was thinking that splitting the system on a country
> wise basis, will allow me to scale easily.

Would each country server be isolated?  Or could two clients in
different countries exchange messages via the servers, like this:

Client -> Server -> Server -> Client

> I am planning to use Amazon EC2,
> so during night times or off peak hours, I'd want the power of different
> instances responsible for different countries to be reduced, to save
> expenses.

That makes sense.  Of course currently EC2 is in three location (2 in
US, 1 in EU).

> Can you point me to real implementations of this kind that have used
> RabbitMQ in the past?

See above.

> Also I'm expecting anywhere between  10-5 M0 Million
> clients, so do you think its feasible?

Yes.

> Also what would be latencies involved
> on an approximate measure...

The main contributor to your latency will be the time taken for data
to travel across the wide area between servers and clients or vice
versa.

alexis

> Regards,
> Karthik Kastury
>
> http://dailyapps.net/ | +91 9739864254
>
>
> On Wed, Dec 30, 2009 at 3:52 PM, Alexis Richardson
> <alexis.richardson at gmail.com> wrote:
>>
>> Karthik
>>
>> Some questions:
>>
>> What is the nature of this communication?  Is it pubsub?  Must queues
>> be involved?  What kind of recovery and failover do you need?
>>
>> Are you saying that the server(s) must be in one physical location?
>>
>> Or, are you saying there must be one server "bot" per country?  If so
>> then can there be communication 'between server bots'?
>>
>> alexis
>>
>>
>> On Wed, Dec 30, 2009 at 2:26 AM, Karthik <coolkarthik88 at gmail.com> wrote:
>> > Hi,
>> >
>> > I am looking for a solution for the problem that am currently looking
>> > to solve... I am trying to build a scalable AMQP Messaging system,
>> > where multiple bots(in the range of 1 for each country in the world)
>> > push out messages to clients spread all over the world. The
>> > communication will be truly one to one, that is one server bot will
>> > communicate with bots, and each client is assigned one server bot.
>> > Messages are mostly private, i.e. only from the server to client and
>> > vice versa.
>> >
>> > There can be anywhere between 10-50 Million Concurrent Clients,
>> > Messages are in the size of 2-3KB(Well formed XML), that can be
>> > transmitted in the both the directions i.e. Client->Server(bots) and
>> > vice versa.... My estimates put at the messages to be going outwards
>> > in either directions somewhere in the range of 50-100 messages/second.
>> >
>> > The Clients will need to send/receive all the information as encrypted
>> > data using SSL+TLS and any other scheme..
>> >
>> > Will RabbitMQ be a good choice for the kind of the problem I'm trying
>> > to solve..
>> >
>> > Any help will be appreciated.. Also how much better is using AMQP
>> > based RabbitMQ in comparison to XMPP based Ejabberd/Openfire etc?
>> > Regards,
>> > Karthik Kastury
>> >
>> > http://dailyapps.net/ | +91 9739864254
>> >
>> > _______________________________________________
>> > rabbitmq-discuss mailing list
>> > rabbitmq-discuss at lists.rabbitmq.com
>> > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> >
>> >
>
>