[rabbitmq-discuss] Scalable multithreaded consumer

Wed Jan 30 13:31:04 GMT 2008

Hi Karel,

Karel Alfonso wrote:
>   There is a Java component in the system that is activated via a scheduled
> task that sends 10 000 events (messages) to start the mail download via POP3
> for each user.

Each message is basically "please poll the POP3 mailbox for user XYZ", then?

> In Java [...] I wished I could have 10 000 threads running at the same time

Oh yes, wouldn't that be nice :-)

>   I'm considering Erlang because of the scalability and concurrency
> features. I know I can have thousands of Erlang process connected to POP3
> accounts on the internet, provided I have enough file descriptors, without
> doing many fancy stuff.

Absolutely. Because of the file-descriptor limit, you'd probably still
want some kind of queueing system in place, though. (Even if it was just
an ad-hoc one on the erlang side.)

>   When I came across RabbitMQ, I thought that was a good way to interface
> Java with Erlang ... But first I wanted to know if I could process or
> consume the messages on the Java side with an efficient thread pool, then in
> a second phase, move the message consumption to Erlang.

Yes, that is a sensible thing to do. A queue containing all the
poll-requests could have several consumers, in one or several JVMs,
consuming from it, making a simple load-balancing worker pool.

How are you getting on? If you're using RabbitMQ, and have any questions
or comments, we'd love to hear about it.

Thanks,
  Tony
-- 
 [][][] Tony Garnock-Jones     | Mob: +44 (0)7905 974 211
   [][] LShift Ltd             | Tel: +44 (0)20 7729 7060
 []  [] http://www.lshift.net/ | Email: tonyg at lshift.net