[rabbitmq-discuss] Scalable multithreaded consumer

[Karel Alfonso]

Hi everyone,

  First, thanks so much for the support and quick replies.

  The problem I have at hand is to "periodically poll the POP3 accounts of 5
thousand users or more and download the mails that match certain criteria".

  As the user base will increase in the future, I'm thinking of having a
clustered solution for scalability purpose.

  Right now I'm using ActiveMQ with clustered queue consumers. The master
distributes the messages in a round-robin manner to each of the consumers.
The consumers handle the message through Jencks, an MDB solution which has a
thread pool. To scale this architecture I have to add more consumer nodes
(with Jenck).

  I thought about Erlang cause in only one box I can have thousands of
processes running, provided that I have enough file descriptors in the OS.
Then if on top of that I can have a clustered solution with RabbitMQ, that
would be the perfect solution for me.

  Using RabbitMQ's Java libraries to consume the messages using a thread
pool does not offer any advantage to me, given that that is the solution I
have at the moment.

  The other question might be out of the subject of this list, but once you
have a RabbitMQ cluster, which technologies would you suggest to share the
file system amongst the cluster?

 Thanks in advance for all your help.

Regards,
 
Karel Alfonso
Tech Lead
Inventri
Level 4, 365 Little Collin St.
Melbourne, VIC 3000
 
Phone: 61 3 86760374
Mobile: 0402663985

-----Original Message-----
From: Tony Garnock-Jones [mailto:tonyg at lshift.net] 
Sent: Friday, 1 February 2008 2:12 AM
To: Karel Alfonso
Cc: rabbitmq-discuss at lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Scalable multithreaded consumer

Hi Karel,

Karel Alfonso wrote:
> In my solution each message means "poll the pop3 account of user XYZ for
new
> mails", that's right. So this is a solution where I need thousands of
> clients/consumers to poll a POP3 account for a specific user.

RabbitMQ's Java library and Erlang broker should have no trouble with
that kind of solution - unless there was some showstopper you've found
that I haven't thought of?

Can you tell us more about the surrounding environment - for instance,
is Spring an important part of your total application?

Regards,
  Tony
-- 
 [][][] Tony Garnock-Jones     | Mob: +44 (0)7905 974 211
   [][] LShift Ltd             | Tel: +44 (0)20 7729 7060
 []  [] http://www.lshift.net/ | Email: tonyg at lshift.net