[rabbitmq-discuss] Fwd: RabbitMQ C library function amqp_simple_wait_frame takes 400 ms
Scott Brooks
scott at beamdog.com
Wed Jun 30 22:14:33 BST 2010
Could you post your sample code, or as simple as possible repro case?
Scott Brooks
On Wed, Jun 30, 2010 at 2:54 PM, Jim Irrer <irrer at umich.edu> wrote:
> What I'm trying to implement is a simple client-server paradigm where each
> is single threaded. The client sends a message to the server, and then
> waits for a reply by posting a read on a separate reply queue. When the
> server gets the message, it processes it, builds a reply, and sends the
> reply to the client.
>
> I've re-built the rabbitmq-c-default library and inserted print statements
> that show the time in microseconds, and the series of events is a loop that
> basically does the following:
>
> server waits on read
>
> client sends message and then waits for reply
>
> server gets message
>
> server performs trivial processing
>
> server sends reply
>
> client gets reply
>
> Below is a listing of events with timestamps in microseconds. The initial
> 3172304
> microsecond delay is the time it took me to start the client in a different
> window
> after starting the server. When a process is "waiting on read", it is
> actually the
> system read call that is posted on the socket. I put the print statement
> inside the
> AMQP library call.
>
> relative elapsed
> time time operation
> -------- ------- ---------------------------------------
> 0 0 server waiting on read
> 3172304 3172304 client calling amqp_basic_publish
> 3172331 27 client returned from amqp_basic_publish
> 3172337 6 client waiting on read
> 3383774 211437 server returned from read
> 3383892 118 server 1 message: 1
> 3383898 6 server calling amqp_basic_publish
> 3383927 29 server returned from amqp_basic_publish
> 3383938 11 server waiting on read
> 3497244 113306 client returned from read
> 3497279 35 client message: 1
> 3497288 9 client calling amqp_basic_publish
> 3497310 22 client returned from amqp_basic_publish
> 3497315 5 client waiting on read
> 3715343 218028 server returned from read
> 3715374 31 server 2 message: 2
> 3715378 4 server calling amqp_basic_publish
> 3715400 22 server returned from amqp_basic_publish
> 3715405 5 server waiting on read
> 3934206 218801 client returned from read
> 3934236 30 client message: 2
> 3934243 7 client calling amqp_basic_publish
> 3934265 22 client returned from amqp_basic_publish
> 3934270 5 client waiting on read
> 4153113 218843 server returned from read
> 4153151 38 server 3 message: 3
> 4153158 7 server calling amqp_basic_publish
> 4153189 31 server returned from amqp_basic_publish
> 4153197 8 server waiting on read
> 4371982 218785 client returned from read
> 4372017 35 client message: 3
> 4372027 10 client calling amqp_basic_publish
> 4372091 64 client returned from amqp_basic_publish
> 4372099 8 client waiting on read
> 4590627 218528 server returned from read
> 4590657 30 server 4 message: 4
> 4590661 4 server calling amqp_basic_publish
> 4590682 21 server returned from amqp_basic_publish
> 4590687 5 server waiting on read
> 4809133 218446 client returned from read
> 4809162 29 client message: 4
> 4809169 7 client calling amqp_basic_publish
> 4809190 21 client returned from amqp_basic_publish
> 4809195 5 client waiting on read
> 5028276 219081 server returned from read
> 5028308 32 server 5 message: 5
> 5028312 4 server calling amqp_basic_publish
> 5028334 22 server returned from amqp_basic_publish
> 5030059 1725 client returned from read
> 5030077 18 client message: 5
>
> Each side seems to be introducing a delay that is suspiciously close to 200
> milliseconds, that may be some system configured value.
>
> I tried TCP_NODELAY (it looks like TCP_NODELACK is only in AIX)
> but it had no effect. Setting this with setsockopt turns off Nagle's
> algorithm which is set by default and tries to optimize socket performance
> for large data transfers by deferring sends until either there is enough
> data to send a full packet or a timeout occurs. Oddly, I had to be root
> to set this option.
>
> If you have any idea as to why the extra delay is being introduced, please
> let me know (and especially how to fix it :) )
>
> Thanks for any insights,
>
> - Jim
>
> Jim Irrer irrer at umich.edu (734) 647-4409
> University of Michigan Hospital Radiation Oncology
> 519 W. William St. Ann Arbor, MI 48103
>
>
> ---------- Forwarded message ----------
> From: Brett Cameron <brett.r.cameron at gmail.com>
> Date: Fri, Jun 25, 2010 at 9:57 PM
> Subject: Re: [rabbitmq-discuss] RabbitMQ C library function
> amqp_simple_wait_frame takes 400 ms
> To: David Wragg <david at rabbitmq.com>, Jim Irrer <irrer at umich.edu>
> Cc: rabbitmq-discuss <rabbitmq-discuss at lists.rabbitmq.com>
>
>
> Jim,
>
> I believe that you are looking at a TCP/IP issue here (which you may or may
> not be able to address by modifying the libRabbitMQ code). My guess is that
> if you set the TCP/IP kernel parameter tcpnodelack (or whatever it is called
> on your operating system) to 1 (i.e. don't delay acknowledging TCP data),
> you will see things improve rather significantly. Depending on what platform
> you're using, you may be able to stick a setsockopt() call (using the option
> TCP_NODELACK) in amqp_socket.c between the socket() call and the connect()
> call instead of having to make the chnage globally by messing with the
> kernel parameter.
> For what it's worth, I encountered this problem with libRabbitMQ-C on
> OpenVMS just last week.. Luky for me I've seen the problem before. Seem to
> recall that you guys at umich.edu used to have some OpenVMS systems...
>
> Regards,
> Brett
>
>
> On Sat, Jun 26, 2010 at 11:54 AM, David Wragg <david at rabbitmq.com> wrote:
>>
>> Hi Jim,
>>
>> Jim Irrer <irrer at umich.edu> writes:
>> > I'm working on two functions that act as a client-server pair. They
>> > use two amq.direct queues to communicate. When ever either of
>> > them calls the amqp_simple_wait_frame function, it does not return
>> > for 436618 microseconds.
>> >
>> > Some other background info that might be relevant:
>> >
>> > If I only send messages in one direction it's really fast.
>> >
>> > Both processes are using separate connectors and different sockets.
>> >
>> > I used the amqp_consumer.c amqp_producer.c code in
>> > the examples directory as a reference.
>> >
>> > Is there a way to avoid this delay?
>>
>> I'm not sure what you are really asking here. As its name suggests,
>> amqp_simple_wait_frame waits for a frame to arrive. It will typically
>> attempt to read from the socket connected to the AMQP server. If no
>> data is available, it will block until data is available. The resulting
>> delays are thus an intrinsic feature of amqp_simple_wait_frame.
>>
>> Are you sure that the 400ms delay does not simply reflect the wait for a
>> message to arrive?
>>
>> I'm guessing, but perhaps the problem is that you want a single
>> application to publish and consume messages concurrently, and you are
>> finding that the synchronous nature of amqp_simple_wait_frame is an
>> obstacle? If so, the simplest work around would be to have two threads,
>> one to publish and one to consume, and open a separate AMQP connection
>> in each thread.
>>
>> > Also ...
>> >
>> > Could I use the same socket in each program as long as it was only
>> > used by one thread at a time?
>> >
>> > Could I use the same connection in each program if it was only
>> > used by one thread at a time?
>>
>> What's the distinction between socket and connection here?
>>
>> librabbitmq does not do anything to explicitly support multithreading,
>> but neither does it do anything to conflict with it. If you, the
>> application programmer, ensure that for a given connection, only one
>> thread uses librabbitmq at a time, you should be safe.
>>
>> --
>> David Wragg
>> Staff Engineer, RabbitMQ
>> SpringSource, a division of VMware
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
More information about the rabbitmq-discuss
mailing list