[rabbitmq-discuss] Scalable multithreaded consumer
tonyg at lshift.net
Wed Jan 30 13:31:04 GMT 2008
Karel Alfonso wrote:
> There is a Java component in the system that is activated via a scheduled
> task that sends 10 000 events (messages) to start the mail download via POP3
> for each user.
Each message is basically "please poll the POP3 mailbox for user XYZ", then?
> In Java [...] I wished I could have 10 000 threads running at the same time
Oh yes, wouldn't that be nice :-)
> I'm considering Erlang because of the scalability and concurrency
> features. I know I can have thousands of Erlang process connected to POP3
> accounts on the internet, provided I have enough file descriptors, without
> doing many fancy stuff.
Absolutely. Because of the file-descriptor limit, you'd probably still
want some kind of queueing system in place, though. (Even if it was just
an ad-hoc one on the erlang side.)
> When I came across RabbitMQ, I thought that was a good way to interface
> Java with Erlang ... But first I wanted to know if I could process or
> consume the messages on the Java side with an efficient thread pool, then in
> a second phase, move the message consumption to Erlang.
Yes, that is a sensible thing to do. A queue containing all the
poll-requests could have several consumers, in one or several JVMs,
consuming from it, making a simple load-balancing worker pool.
How are you getting on? If you're using RabbitMQ, and have any questions
or comments, we'd love to hear about it.
 Tony Garnock-Jones | Mob: +44 (0)7905 974 211
 LShift Ltd | Tel: +44 (0)20 7729 7060
  http://www.lshift.net/ | Email: tonyg at lshift.net
More information about the rabbitmq-discuss