Matthias,<div>Attached is a zipped Visual Studio 2008 solution. This exhibits the problems I am seeing. The code contains a producer.exe and consumer.exe, both projects share a sharedTypes.dll. This all builds from one solution once you update the reference to a RabbitMQ .NET Client. The code assumes that the broker is installed locally. Comments are in program.cs of the producer.exe project.</div>
<div><br></div><div>Thanks for taking a look.</div><div><br></div><div>Mark <br><br><div class="gmail_quote">On Mon, May 25, 2009 at 4:07 PM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@lshift.net">matthias@lshift.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Mark,<br>
<br>
smittycb10 wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
however I still have the same problem 100,000's messages in and only<br>
1,000's of messages out. Am I publishing and subscribing in the most<br>
efficent manner?<br>
</blockquote>
<br>
Can you send through the *complete* code? It's impossible to tell<br>
otherwise why exactly you are getting the results you are seeing.<br>
<br>
How are you measuring the rates? If on the sending side you are just<br>
timing a loop of publishes then all you are measuring is the time it<br>
takes to hand over the messages to the kernel. They can get buffered<br>
there, in the network layer, in multiple layers at the broker, etc, etc.<br>
<br>
Also, since AMQP's basic.publish is asynchronous, if the publishing connection is not terminated cleanly then some messages will inevitably get lost.<br>
<br>
It's also worth remembering that RabbitMQ is a message *queuing* system, so one shouldn't be surprised if occasionally it queues some messages ;) What's more, because RabbitMQ's queuing is quite efficient, when you push the system hard, i.e. by feeding it messages as fast as you can, it will handle the incoming messages at a faster rate than it sends them to consumers.<br>
<br>
For meaningful benchmark results it is therefore important to throttle<br>
the producer, e.g. send messages at a fixed rate, monitor the rate at<br>
which they are received by consumers, and slowly increase the producing<br>
rate until the consumption rate starts falling behind.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Do I need to anything to broker?<br>
</blockquote></blockquote>
<br>
Unlikely.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Is a message rate of 100,000 per second unreasonable?<br>
</blockquote></blockquote>
<br>
For a single producer->broker->consumer chain that would be quite a high<br>
rate - 10s of thousands of msgs/s is more realistic on stock hardware.<br>
<br>
If you have several independent message streams then higher total<br>
throughputs can be achieved when running on lots of cores.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
is there a max number of messages any one channel can have queued in<br>
it at one time?<br>
</blockquote>
<br>
No.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I am currently trying to squeeze all the messages into one channel,<br>
should I be splitting work across channels and threads on the<br>
receiving side??<br>
</blockquote>
<br>
Having multiple channels and, even better, multiple connections, provides more opportunities for parellelism at the server. Whether or not that actually makes a positive difference depends on lots of factors.<br>
<br>
<br>
Regards,<br><font color="#888888">
<br>
Matthias.<br>
</font></blockquote></div><br><br clear="all"><br>-- <br>Mark Smith<br><br>please send all correspondences to this <br>new email address<br>
</div>