[rabbitmq-discuss] Fwd: RabbitMQ C library function amqp_simple_wait_frame takes 400 ms

Wed Jun 30 22:14:33 BST 2010

Could you post your sample code, or as simple as possible repro case?

Scott Brooks

On Wed, Jun 30, 2010 at 2:54 PM, Jim Irrer <irrer at umich.edu> wrote:
> What I'm trying to implement is a simple client-server paradigm where each
> is single threaded.  The client sends a message to the server, and then
> waits for a reply by posting a read on a separate reply queue.  When the
> server gets the message, it processes it, builds a reply, and sends the
> reply to the client.
>
> I've re-built the rabbitmq-c-default library and inserted print statements
> that show the time in microseconds, and the series of events is a loop that
> basically does the following:
>
> server waits on read
>
> client sends message and then waits for reply
>
> server gets message
>
> server performs trivial processing
>
> server sends reply
>
> client gets reply
>
> Below is a listing of events with timestamps in microseconds.  The initial
> 3172304
> microsecond delay is the time it took me to start the client in a different
> window
> after starting the server.  When a process is "waiting on read", it is
> actually the
> system read call that is posted on the socket.  I put the print statement
> inside the
> AMQP library call.
>
> relative   elapsed
>   time      time    operation
> --------   -------  ---------------------------------------
>        0         0  server waiting on read
>  3172304   3172304  client calling amqp_basic_publish
>  3172331        27  client returned from amqp_basic_publish
>  3172337         6  client waiting on read
>  3383774    211437  server returned from read
>  3383892       118  server 1 message: 1
>  3383898         6  server calling amqp_basic_publish
>  3383927        29  server returned from amqp_basic_publish
>  3383938        11  server waiting on read
>  3497244    113306  client returned from read
>  3497279        35  client message: 1
>  3497288         9  client calling amqp_basic_publish
>  3497310        22  client returned from amqp_basic_publish
>  3497315         5  client waiting on read
>  3715343    218028  server returned from read
>  3715374        31  server 2 message: 2
>  3715378         4  server calling amqp_basic_publish
>  3715400        22  server returned from amqp_basic_publish
>  3715405         5  server waiting on read
>  3934206    218801  client returned from read
>  3934236        30  client message: 2
>  3934243         7  client calling amqp_basic_publish
>  3934265        22  client returned from amqp_basic_publish
>  3934270         5  client waiting on read
>  4153113    218843  server returned from read
>  4153151        38  server 3 message: 3
>  4153158         7  server calling amqp_basic_publish
>  4153189        31  server returned from amqp_basic_publish
>  4153197         8  server waiting on read
>  4371982    218785  client returned from read
>  4372017        35  client message: 3
>  4372027        10  client calling amqp_basic_publish
>  4372091        64  client returned from amqp_basic_publish
>  4372099         8  client waiting on read
>  4590627    218528  server returned from read
>  4590657        30  server 4 message: 4
>  4590661         4  server calling amqp_basic_publish
>  4590682        21  server returned from amqp_basic_publish
>  4590687         5  server waiting on read
>  4809133    218446  client returned from read
>  4809162        29  client message: 4
>  4809169         7  client calling amqp_basic_publish
>  4809190        21  client returned from amqp_basic_publish
>  4809195         5  client waiting on read
>  5028276    219081  server returned from read
>  5028308        32  server 5 message: 5
>  5028312         4  server calling amqp_basic_publish
>  5028334        22  server returned from amqp_basic_publish
>  5030059      1725  client returned from read
>  5030077        18  client message: 5
>
> Each side seems to be introducing a delay that is suspiciously close to 200
> milliseconds, that may be some system configured value.
>
> I tried TCP_NODELAY (it looks like TCP_NODELACK is only in AIX)
> but it had no effect.  Setting this with setsockopt turns off Nagle's
> algorithm which is set by default and tries to optimize socket performance
> for large data transfers by deferring sends until either there is enough
> data to send a full packet or a timeout occurs.  Oddly, I had to be root
> to set this option.
>
> If you have any idea as to why the extra delay is being introduced, please
> let me know (and especially how to fix it :)  )
>
> Thanks for any insights,
>
> - Jim
>
> Jim Irrer     irrer at umich.edu       (734) 647-4409
> University of Michigan Hospital Radiation Oncology
> 519 W. William St.             Ann Arbor, MI 48103
>
>
> ---------- Forwarded message ----------
> From: Brett Cameron <brett.r.cameron at gmail.com>
> Date: Fri, Jun 25, 2010 at 9:57 PM
> Subject: Re: [rabbitmq-discuss] RabbitMQ C library function
> amqp_simple_wait_frame takes 400 ms
> To: David Wragg <david at rabbitmq.com>, Jim Irrer <irrer at umich.edu>
> Cc: rabbitmq-discuss <rabbitmq-discuss at lists.rabbitmq.com>
>
>
> Jim,
>
> I believe that you are looking at a TCP/IP issue here (which you may or may
> not be able to address by modifying the libRabbitMQ code). My guess is that
> if you set the TCP/IP kernel parameter tcpnodelack (or whatever it is called
> on your operating system) to 1 (i.e. don't delay acknowledging TCP data),
> you will see things improve rather significantly. Depending on what platform
> you're using, you may be able to stick a setsockopt() call (using the option
> TCP_NODELACK) in amqp_socket.c between the socket() call and the connect()
> call instead of having to make the chnage globally by messing with the
> kernel parameter.
> For what it's worth, I encountered this problem with libRabbitMQ-C on
> OpenVMS just last week.. Luky for me I've seen the problem before. Seem to
> recall that you guys at umich.edu used to have some OpenVMS systems...
>
> Regards,
> Brett
>
>
> On Sat, Jun 26, 2010 at 11:54 AM, David Wragg <david at rabbitmq.com> wrote:
>>
>> Hi Jim,
>>
>> Jim Irrer <irrer at umich.edu> writes:
>> > I'm working on two functions that act as a client-server pair.  They
>> > use two amq.direct queues to communicate.  When ever either of
>> > them calls the amqp_simple_wait_frame function, it does not return
>> > for 436618 microseconds.
>> >
>> > Some other background info that might be relevant:
>> >
>> > If I only send messages in one direction it's really fast.
>> >
>> > Both processes are using separate connectors and different sockets.
>> >
>> > I used the amqp_consumer.c amqp_producer.c code in
>> > the examples directory as a reference.
>> >
>> > Is there a way to avoid this delay?
>>
>> I'm not sure what you are really asking here.  As its name suggests,
>> amqp_simple_wait_frame waits for a frame to arrive.  It will typically
>> attempt to read from the socket connected to the AMQP server.  If no
>> data is available, it will block until data is available.  The resulting
>> delays are thus an intrinsic feature of amqp_simple_wait_frame.
>>
>> Are you sure that the 400ms delay does not simply reflect the wait for a
>> message to arrive?
>>
>> I'm guessing, but perhaps the problem is that you want a single
>> application to publish and consume messages concurrently, and you are
>> finding that the synchronous nature of amqp_simple_wait_frame is an
>> obstacle?  If so, the simplest work around would be to have two threads,
>> one to publish and one to consume, and open a separate AMQP connection
>> in each thread.
>>
>> > Also ...
>> >
>> > Could I use the same socket in each program as long as it was only
>> > used by one thread at a time?
>> >
>> > Could I use the same connection in each program if it was only
>> > used by one thread at a time?
>>
>> What's the distinction between socket and connection here?
>>
>> librabbitmq does not do anything to explicitly support multithreading,
>> but neither does it do anything to conflict with it.  If you, the
>> application programmer, ensure that for a given connection, only one
>> thread uses librabbitmq at a time, you should be safe.
>>
>> --
>> David Wragg
>> Staff Engineer, RabbitMQ
>> SpringSource, a division of VMware
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>