[rabbitmq-discuss] Scalable multithreaded consumer

[Karel Alfonso]

Hi everyone,

  Thanks so much for your reply.

  I'm working on a system that requires to download via POP3 every X amount
of time, the email of potentially 10 000 users (each user can have their
POP3 account on the internet).

  There is a Java component in the system that is activated via a scheduled
task that sends 10 000 events (messages) to start the mail download via POP3
for each user.

  In Java we have the problems with the scalability using multithreading,
that is, I wished I could have 10 000 threads running at the same time on
the system. I'm also limited by the number of File descriptors the OS will
allow me to open at once and the number of threads per process, etc.

  I'm considering Erlang because of the scalability and concurrency
features. I know I can have thousands of Erlang process connected to POP3
accounts on the internet, provided I have enough file descriptors, without
doing many fancy stuff.

  When I came across RabbitMQ, I thought that was a good way to interface
Java with Erlang ... But first I wanted to know if I could process or
consume the messages on the Java side with an efficient thread pool, then in
a second phase, move the message consumption to Erlang.

  I hope that gives a better explanation of what I'm after. It might be the
case the RabbitMQ is not the right technology for what I'm trying to do. I
appreciate your time and suggestions

Regards,
 
Karel Alfonso
Tech Lead
Inventri
Level 4, 365 Little Collin St.
Melbourne, VIC 3000
 
Phone: 61 3 86760374
Mobile: 0402663985
-----Original Message-----
From: Matthias Radestock [mailto:matthias at lshift.net] 
Sent: Monday, 21 January 2008 7:00 AM
To: Karel Alfonso
Cc: rabbitmq-discuss at lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Scalable multithreaded consumer

Karel,

Karel Alfonso wrote:

>  I'm planning to use RabbitMQ in a system that will process events in 
> the order of the thousands. I haven't found in RabbitMQ a mechanism to 
> process concurrently the messages received. Is there any suggestion or 
> guideline I could follow?

It is certainly possible to perform high-volume concurrent event 
processing with RabbitMQ, but coming up with the best setup requires a 
better understanding of your specific requirements.

Can you elaborate on what you are trying to accomplish? How many event 
producers are there? How many event consumers are there? What routing do 
you need between them? What delivery guarantees do you require? What 
ordering guarantees do you require? What AMQP client libraries are you 
using (e.g. RabbitMQ's Java client, the Erlang client etc)?


Regards,

Matthias.