[rabbitmq-discuss] Scalable multithreaded consumer

Wed Jan 30 21:55:29 GMT 2008

Thanks Tony for your reply.

I have to say I decided to use ActiveMQ with clustered queue consumers where
each of them has a thread pool managed by Jencks, an MDB solution that
integrates nicely with ActiveMQ. 

I would have loved to have RabbitMQ and Erlang consumers but I have to wait
till we have Erlang client implemented in RabbitMQ, then we can move away
from consuming messages in Java.

In my solution each message means "poll the pop3 account of user XYZ for new
mails", that's right. So this is a solution where I need thousands of
clients/consumers to poll a POP3 account for a specific user.

I looked into an Erlang solution without RabbitMQ, but I don't think
enterprise APIs in Erlang are as mature and proven as those for Java. For
example, the POP3 client in Erlang, epop, dates back since 1998, and we
don't have yet a JavaMail-like package in Erlang, so, things become more
difficult.

Scala offeres the Acto model as well, but using the threading capability
offered by Java.

For the moment I'll use ActiveMQ with clustered consumers, in the future
I'll move the implementation to use RabbitMQ with Erlang consumers.

Thanks for your support,

  Cheers,

  Karel

-----Original Message-----
From: Tony Garnock-Jones [mailto:tonyg at lshift.net] 
Sent: Thursday, 31 January 2008 12:31 AM
To: Karel Alfonso
Cc: rabbitmq-discuss at lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Scalable multithreaded consumer

Hi Karel,

Karel Alfonso wrote:
>   There is a Java component in the system that is activated via a
scheduled
> task that sends 10 000 events (messages) to start the mail download via
POP3
> for each user.

Each message is basically "please poll the POP3 mailbox for user XYZ", then?

> In Java [...] I wished I could have 10 000 threads running at the same
time

Oh yes, wouldn't that be nice :-)

>   I'm considering Erlang because of the scalability and concurrency
> features. I know I can have thousands of Erlang process connected to POP3
> accounts on the internet, provided I have enough file descriptors, without
> doing many fancy stuff.

Absolutely. Because of the file-descriptor limit, you'd probably still
want some kind of queueing system in place, though. (Even if it was just
an ad-hoc one on the erlang side.)

>   When I came across RabbitMQ, I thought that was a good way to interface
> Java with Erlang ... But first I wanted to know if I could process or
> consume the messages on the Java side with an efficient thread pool, then
in
> a second phase, move the message consumption to Erlang.

Yes, that is a sensible thing to do. A queue containing all the
poll-requests could have several consumers, in one or several JVMs,
consuming from it, making a simple load-balancing worker pool.

How are you getting on? If you're using RabbitMQ, and have any questions
or comments, we'd love to hear about it.

Thanks,
  Tony
-- 
 [][][] Tony Garnock-Jones     | Mob: +44 (0)7905 974 211
   [][] LShift Ltd             | Tel: +44 (0)20 7729 7060
 []  [] http://www.lshift.net/ | Email: tonyg at lshift.net