[rabbitmq-discuss] Will this work?

Thu Jan 21 11:28:36 GMT 2010

Hi Tony,

Thanks for your reply. 

I tried disabling Nagle, but it does really not help for cases a & b. Moreover, it messes up latency for higher rates.

I tried googling about Nagle, and found a interesting article (http://www.stuartcheshire.org/papers/NagleDelayedAck/) that mentions that there is a possibility of some kind of interaction between nagle and delayed ack that can introduce delays of around 200 ms. The latency figures in cases 1 & 2 are very close to that figure.

>From that article it seems an alternative way (other than disabling nagle) is to have a two way communication between the peers of a socket, i.e if the peer has something to send then the ack will not be delayed. In my test application the communication is always one way, either from application to broker (publisher) or from broker to application (consumer). To have a two way communication I need have two threads (consumer & publisher) sharing the same socket.

I searched the mailing list for correct way to write a multithreaded application for RabbitMQ and found a post from you that suggests I should have a single shared socket but thread specific connection and channel.

But I landed into problem trying doing it. The problem is if I create two connections on the same socket and attempt a login on the second connection, the application simply dies (no core). I simulated this in a trivial code. The code is below -

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#include <stdint.h>
#include <amqp.h>
#include <amqp_framing.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <netinet/tcp.h>

int main()
{
    int socket = amqp_open_socket("127.0.0.1", 5672);

    amqp_connection_state_t connection1 = amqp_new_connection();
    amqp_set_sockfd(connection1, socket);
    amqp_login(connection1, "/", 0, 131072, 0, AMQP_SASL_METHOD_PLAIN, "guest", "guest");
    amqp_channel_open(connection1, 1);

    printf ("Connection 1 .... ok\n");

    amqp_connection_state_t connection2 = amqp_new_connection();
    amqp_set_sockfd(connection2, socket);
    printf ("Login on Connection 2 ....\n"); /* This print comes */
    amqp_login(connection2, "/", 0, 131072, 0, AMQP_SASL_METHOD_PLAIN, "guest", "guest"); /* Application simply dies here ... no core ...nothing */
    printf ("Login on Connection 2 .... ok\n");  /* This print does not come */
    amqp_channel_open(connection2, 1);

    printf ("Connection 2 .... ok\n");
} 

Can you please help on where I am going wrong?

Regards,
Dinabandhu

> -----Original Message-----
> From: Tony Garnock-Jones [mailto:tonyg at lshift.net]
> Sent: Wednesday, January 20, 2010 1:27 AM
> To: Dinabandhu Mitra
> Cc: Alexis Richardson; rabbitmq info; rabbitmq-
> discuss at lists.rabbitmq.com
> Subject: Re: [rabbitmq-discuss] Will this work?
> 
> Hi Dinabandhu,
> 
> I think you're running into TCP_NODELAY issues. Try setting your
> client's socket to nodelay mode, which disables Nagling:
> 
>   int one = 1;
>   setsockopt(sockfd, IPPROTO_TCP, TCP_NODELAY, &one, sizeof(one));
> 
> This should fix the high observed latency in cases (a) and (b).
> 
> Regards,
>   Tony
> 
> 
> 
> Dinabandhu Mitra wrote:
> > Hi Alexis,
> >
> > Any further advice on our earlier discussions?
> >
> >
> > I am currently trying out using c API. I was trying to measure
> communication latency introduced by RabbitMQ. Following are some of the
> observations -
> >
> > 1. When the test application starts, the latency for first few
> messages is very high. The number of messages with high latency depends
> on the throughput the application tries to generate. E.g -
> > 	a) For 1/2 message per sec latency continues to be very high
> (over 200 millisecond) throughout.
> > 	b) For 3/4 message per sec it seems to change between high (200)
> and low (0.200) on every alternate message.
> > 	c) For 5 messages per sec it stabilizes to a low value (0.200 -
> 0.300) after first 5/6 messages.
> > 	d) For 10 messages per sec it stabilizes to a low value (0.200 -
> 0.300) after first 5/6 messages.
> > 	e) For 50 messages per sec it stabilizes to a low value (0.200 -
> 0.300) after first 15/16 messages.
> > 	f) For 100 messages per sec it stabilizes to a low value (0.200 -
> 0.300) after first 50/60 messages.
> > 	g) For 500 messages per sec it stabilizes to a low value (0.300 -
> 0.500) after first 370/380 messages.
> > 	h) For 1000 messages per sec it comes down to a low value (0.500
> - 0.700) after first 500 messages. But I have seen intermittent bursts
> of a sequence of high latency (going over 20 millisecond) messages.
> > 	i) Initial latencies increase with throughput. E.g at 1000
> msg/sec it is as high as 380/390 millisecond for some messages.
> >
> > 2. Both publisher and subscriber are in the same server as the
> broker. In fact the publisher and the subscriber are two threads in the
> same process.
> >
> > 3. The server is a idle SUN Netra Dual CPU(Quad Core) Xeon class
> machine with 16G RAM running RHEL 4.7 64 bit.
> >
> > 4. The platform is on the factory configuration; i.e. I have just
> installed the RPMs and started the RabbitMQ server without any other
> configuration.
> >
> > 5. I am using approx 1KB sized messages for this.
> >
> > Is there any configuration that I can do to have better (less
> variance) latency? Also, I am using 1 producer and 1 consumer (both
> trivial). Shall I get better results by using multiple
> producers/consumers?
> >
> > Regards,
> > Dinabandhu
> >
> >> -----Original Message-----
> >> From: Alexis Richardson [mailto:alexis.richardson at gmail.com]
> >> Sent: Tuesday, January 05, 2010 10:33 PM
> >> To: Dinabandhu Mitra
> >> Cc: rabbitmq info
> >> Subject: Re: [rabbitmq-discuss] Will this work?
> >>
> >> Dinabandhu
> >>
> >> Answers / comments below also.  Let me know if you want to cc this
> to
> >> rabbitmq-discuss.  FYI - info at rabbitmq.com is our private (internal
> >> team) list.
> >>
> >> alexis
> >>
> >>
> >>
> >> On Tue, Jan 5, 2010 at 3:16 PM, Dinabandhu Mitra
> >> <Dinabandhu.Mitra at tecnotree.com> wrote:
> >>> The scale of the application depends on the implementation. The
> >> messaging load
> >>> can vary from few hundred per second to over 30,000/40,000 per
> >> second. Of course
> >>> we will add additional equipment depending on the load but the
> >> application should
> >>> be able to use additional equipment effectively.
> >> OK.  That should be fine.  What's the peak ingress per node?
> >>
> >> I.e. what is the max rate that a single publisher might produce in
> >> messages per second?
> >>
> >> Also - are the messages 'small' (< 500 bytes) or 'large' (>64Kb)?
> Is
> >> this size consistent or volatile?
> >>
> >>
> >>
> >>
> >>>> OK.  You will be able to do this with direct exchanges too.  They
> >> are
> >>>> faster (in lookup time) than our current topic implementation.  It
> >>>> does not look like you need topic exchanges because you do not
> >> appear
> >>>> to be routing messages using wildcards, eg. you do not need
> bindings
> >>>> of the form "PE.*".  Is that right?
> >>> Yes. I was reading the AMQP specification concepts section few days
> >> back
> >>> and realized that my understanding of direct exchanges was wrong.
> >> Direct
> >>> exchange should be the right option for us.
> >> Good.
> >>
> >>
> >>
> >>
> >>>>>   a) When a application engine (LE/PE) boots and declares it's
> >> queue
> >>>> and
> >>>>> binding using a connection to the local broker, the queue and
> >>>> bindings
> >>>>> becomes visible to all broker instances running in different
> >> servers.
> >>>> Is
> >>>>> this correct?
> >>>> Correct, provided that the other brokers are in the same cluster
> (as
> >>>> in your diagram) and vhost (ie. namespace).
> >>> It should be sufficient. As of now we are looking at a single
> cluster
> >> and
> >>> single vhost.
> >> OK, cool.
> >>
> >>
> >>
> >>>>>   f) Anyone trying send a message that is immediately non-
> routable
> >>>>> (possibly mandatory+immediate flag with auto-ack) gets a error
> and
> >>>> broker
> >>>>> drops the message. Is this correct?
> >>>> I can't remember exactly how this works.  Basically the answer is
> >> YES.
> >>>>  I cannot recall under what conditions the broker drops the
> message
> >>>> silently, and under what conditions it lets the publisher know
> that
> >> an
> >>>> error occurred.
> >>> Well, this is not a very hard requirement. The LE instances are
> timer
> >> controlled,
> >>> in the sense that if a LE does not receive a response from any of
> the
> >> PEs within
> >>> certain time limit then the LE assumes that there is a problem is
> >> handling the
> >>> request and sends a failure response to the client that originated
> >> the transaction.
> >>> However, it is way better if the rejection can be done without
> >> waiting for the
> >>> timer to expire in case none of the relevant PE engines are live.
> >> This is how the
> >>> application behaves currently.
> >> OK.  We can return to this point later.
> >>
> >> alexis
> >>
> >>
> >>
> >>>> alexis
> >>>>
> >>>>
> >>>>
> >>>>> Regards,
> >>>>> Dinabandhu
> >>>>> --
> >>>>> View this message in context: http://old.nabble.com/Will-this-
> >> work--
> >>>> tp26875409p26875409.html
> >>>>> Sent from the RabbitMQ mailing list archive at Nabble.com.
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> rabbitmq-discuss mailing list
> >>>>> rabbitmq-discuss at lists.rabbitmq.com
> >>>>> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-
> >> discuss
> >