[rabbitmq-discuss] Scalability?

Fri May 7 06:51:02 BST 2010

I second what Colin says,

We use RabbitMQ in prod for nearly a year now, and we never had problems. Our volumes are not big, but we queue/consume around 500K messages per day.

We only stopped the servers when we upgraded them to the latest version,

Regards,

Alvaro

On May 7, 2010, at 6:08 AM, Jason J. W. Williams wrote:

> Hi Wayne,
> 
> This there a reason you're using basic_get instead of basic_consume? I
> haven't looked at the guts of basic_get inside py-amqplib, but
> basic_consume just by my way of thinking should have less
> build-up/tear-down overhead. It's what we use very successfully.
> 
> -J
> 
> On Thu, May 6, 2010 at 11:49 AM, Wayne Van Den Handel
> <wvandenhandel at dataraker.com> wrote:
>> I am evaluating RabbitMQ for purposes of parallelization on top of a
>> Cassandra data store. I have created a simple test scenario of a set of
>> Queues that are given data to be loaded from a single Python publisher
>> and 3-4 Python Consumer applications take the data from the Queues and
>> load into Cassandra. The entire scenario was easily set up and runs
>> great for about 10 minutes when RabbitMQ proceeds to use up all
>> available memory and crashes.  I then discovered the passive mode to
>> create a queue (and find out how many messages it has) and now only add
>> more work to the queue when there is less than 1000 messages in the
>> queue (which easily fit into memory). I start up my test again and still
>> blow RabbitMQ up in 10 minutes. I am watching with the admin console the
>> entire time and there is never more than a total 1000 messages in all
>> queues at any given time. Watching top I see RabbitMQ take up more and
>> more memory over time. It seems that it can only process 30-40k messages
>> in total/aggregate before it crashes (even though there is never more
>> than 1000 messages in all queues at one time).
>> 
>> Am I missing something here? The product seems very easy to use and
>> works great but it totally un-scalable. Is RabbitMQ not meant for high
>> data volumes/traffic? What would better serve this purpose? We need
>> something on top of Cassandra to provide high volume parallelization. I
>> understand that we can only hold what fits in memory right now (when
>> will that be fixed?), but even that is not true as memory is never given
>> back.
>> 
>> Environment:
>> CentOS 5.4 64 Bit
>> RabbitMQ v1.7.2-1.el5 installed from yum
>> py-amqplib
>> 
>> Create Queue
>> chan.queue_declare(queue="dr_load.1", durable=True, exclusive=False,
>> auto_delete=False)
>> chan.exchange_declare(exchange="dr_load", type="direct", durable=True,
>> auto_delete=False)
>> chan.queue_bind(queue="dr_load.1", exchange="dr_load",
>> routing_key="Instance.1")
>> 
>> Publish Data
>> chan.basic_publish(msg,exchange="dr_load",routing_key="Instance.1",mandatory=True)
>> 
>> Consume Data
>> msg = chan.basic_get("Instance.1")
>> chan.basic_ack(msg.delivery_tag)
>> 
>> Thanks!
>> 
>> --
>> Wayne Van Den Handel, DataRaker Inc
>> 
>> Phone:  703.996.4891
>> Mobile: 305.849.1794
>> Skype:  wayne.van.den.handel
>> Email:  wvandenhandel at dataraker.com
>> 
>> 
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> 
> 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss