[rabbitmq-discuss] Pika Question (publish and consume as fast as possible)

[Aaron Westendorf]

I've documented to the list and also on our blog that I've been able
to use 3 hosts, each running 3 python processes, to push 40-48k
messages/sec through a single broker, which is in the 4-5k/sec per
process range. The stress test uses direct exchanges and the hosts are
VMs.

With Python and the GIL, there are practical limits to what a single
process is capable of, but it is still very fast given the following
conditions:
* IO is as close to the C implementation as possible
* Allocations per message cycle are deterministic and small

I've had great experience with IO in all Python applications I've
written. As always, vigilance and a keen eye make all the difference.

-Aaron


On Mon, Jul 18, 2011 at 11:06 AM, Marek Majkowski <majek04 at gmail.com> wrote:
> On Mon, Jul 18, 2011 at 14:39, Josh Geisser <josh at gebaschtel.ch> wrote:
>> Anyway, I want to write a little AMPQ Ping tool which should allow us to test various delivery patterns over a Rabbit Cluster.
>>
>> For that I want to have the possibility to publish and consume as fast as possible, also kind of benchmarking the system.
>
> You are aware that Python is not the fastest language, right?
>
> My rough guess would be that for a decent client you should be able to
> get something like 1k - 6k msgs/sec for simple messaging patterns.
>
>>
>> I made some progress in getting into Pika and think I understand the idea of the TCP_Backpressure, but I find it a bit hard to implement it on the 'enduser' side. (I found the old idea of TX_ and TX_commit easier to prevent overflow...)
>>
>> Long speech short, I ended up with using 'BlockingConnection' for publishing and 'SelectConnection' for consuming, otherwise i always ran into buffer-warnings and eventually crashes ....
>>
>> Could someone provide me with a quick snipped that publish/consumes as fast as possible, the way you'd implement it?
>>
>> Cheers & thanks a lot
>> Josh
>>
>> (pika in my case latest git-version, Python 2.7.2)
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>



-- 
Aaron Westendorf
Senior Software Engineer
Agora Games
359 Broadway
Troy, NY 12180
Phone: 518.268.1000
aaron at agoragames.com
www.agoragames.com