[rabbitmq-discuss] Performance and scalability measurement with my publisher-subscriber clients testing

[Marek Majkowski]

On Wed, Mar 16, 2011 at 12:06, Deepak Vijayvergiy <vdeepak at yahoo-inc.com> wrote:
> Thanks Marek for the good points.
> I am thinking of having 1 publisher and a few consumers consuming from the same queue, so that I don't get queue full (as publisher will be faster than subscriber).
> In that scenario, I will try to publish messages continuously and see the rate of publisher as well as consumer, and while mentioning the performance numbers, will give the setup details as well.
>
> Also, another question regarding the rabbitmq queue space, I see that its limited to 1.58 GB, is that the size on the  disk by default or its the RAM size?
> How shall I configure it if I want to increase it?

You're probably speaking about Memory Limit line in your logs:
  http://www.rabbitmq.com/extensions.html#memsup

That indicates that Rabbit will try to keep its RAM usage below the value.
In the docs you will see how to increase this value.

It doesn't have much to do with the maximum queue size.

Marek

> On Mar 16, 2011, at 5:26 PM, Marek Majkowski wrote:
>
>> On Wed, Mar 9, 2011 at 10:26, Deepak Vijayvergiy <vdeepak at yahoo-inc.com> wrote:
>>> I have created the framework for publisher-subscriber and created publisher and consumer clients for basic testing as well. Now, I also want to test the performance as well, as in, I want to give the numbers like -
>>> (1) How many messages, my publisher client will be able to publish per second or rate?
>>> (2) How many messages my consumer will be able to consume (per second) .. rate? (a) with messages already in the queue (b) parallely publishing as well
>>>
>>> Earlier I have tried doing same by some time functions before and after 1 and last mess respectively, but I don't think that was a proper way. Please let me know if you have done the same measurement?
>>
>> We do have a tool for doing simple measurements, see MulticastMain
>> in our java client.  Sample usage:
>>   http://www.rabbitmq.com/ec2.html#testing
>> (dig the mailing list archives for more examples)
>>
>> Doing benchmarks properly, even as simple as you described, is pretty hard.
>>
>> With 1) you will measure performance of the client. Unless you're using
>> publisher-acks.
>>
>> 2) It all depends, if data in-memory or on the disk, is it cached by the OS,
>> how big is the queue, how many queues are handled by rabbit and so on.
>>
>> 2b) That one is quite interesting, but you have to be very careful to keep
>> the queue balanced. Publishing is usually faster than consuming.
>>
>> Rabbit is not a rocket science and there are some significant bottlenecks.
>> If you start swapping to disk or if you use persistent messages -
>> things will be much slower.
>>
>> Usually the best approach is to test the whole solution end-to-end
>> rather than focusing on dry RabbitMQ benchmarks. That way you can
>> be sure that the numbers actually mean something you're interested
>> in. Raw benchmarks like publishes/sec or consumes/sec doesn't
>> say anything useful.
>>
>> Cheers,
>>  Marek
>
>