[rabbitmq-discuss] Fwd: RabbitMQ C library function amqp_simple_wait_frame takes 400 ms
Jim Irrer
irrer at umich.edu
Wed Jun 30 21:54:55 BST 2010
What I'm trying to implement is a simple client-server paradigm where each
is single threaded. The client sends a message to the server, and then
waits for a reply by posting a read on a separate reply queue. When the
server gets the message, it processes it, builds a reply, and sends the
reply to the client.
I've re-built the rabbitmq-c-default library and inserted print statements
that show the time in microseconds, and the series of events is a loop that
basically does the following:
server waits on read
client sends message and then waits for reply
server gets message
server performs trivial processing
server sends reply
client gets reply
Below is a listing of events with timestamps in microseconds. The initial
3172304
microsecond delay is the time it took me to start the client in a different
window
after starting the server. When a process is "waiting on read", it is
actually the
system read call that is posted on the socket. I put the print statement
inside the
AMQP library call.
relative elapsed
time time operation
-------- ------- ---------------------------------------
0 0 server waiting on read
3172304 3172304 client calling amqp_basic_publish
3172331 27 client returned from amqp_basic_publish
3172337 6 client waiting on read
3383774 211437 server returned from read
3383892 118 server 1 message: 1
3383898 6 server calling amqp_basic_publish
3383927 29 server returned from amqp_basic_publish
3383938 11 server waiting on read
3497244 113306 client returned from read
3497279 35 client message: 1
3497288 9 client calling amqp_basic_publish
3497310 22 client returned from amqp_basic_publish
3497315 5 client waiting on read
3715343 218028 server returned from read
3715374 31 server 2 message: 2
3715378 4 server calling amqp_basic_publish
3715400 22 server returned from amqp_basic_publish
3715405 5 server waiting on read
3934206 218801 client returned from read
3934236 30 client message: 2
3934243 7 client calling amqp_basic_publish
3934265 22 client returned from amqp_basic_publish
3934270 5 client waiting on read
4153113 218843 server returned from read
4153151 38 server 3 message: 3
4153158 7 server calling amqp_basic_publish
4153189 31 server returned from amqp_basic_publish
4153197 8 server waiting on read
4371982 218785 client returned from read
4372017 35 client message: 3
4372027 10 client calling amqp_basic_publish
4372091 64 client returned from amqp_basic_publish
4372099 8 client waiting on read
4590627 218528 server returned from read
4590657 30 server 4 message: 4
4590661 4 server calling amqp_basic_publish
4590682 21 server returned from amqp_basic_publish
4590687 5 server waiting on read
4809133 218446 client returned from read
4809162 29 client message: 4
4809169 7 client calling amqp_basic_publish
4809190 21 client returned from amqp_basic_publish
4809195 5 client waiting on read
5028276 219081 server returned from read
5028308 32 server 5 message: 5
5028312 4 server calling amqp_basic_publish
5028334 22 server returned from amqp_basic_publish
5030059 1725 client returned from read
5030077 18 client message: 5
Each side seems to be introducing a delay that is suspiciously close to 200
milliseconds, that may be some system configured value.
I tried TCP_NODELAY (it looks like TCP_NODELACK is only in AIX)
but it had no effect. Setting this with setsockopt turns off Nagle's
algorithm which is set by default and tries to optimize socket performance
for large data transfers by deferring sends until either there is enough
data to send a full packet or a timeout occurs. Oddly, I had to be root
to set this option.
If you have any idea as to why the extra delay is being introduced, please
let me know (and especially how to fix it :) )
Thanks for any insights,
- Jim
Jim Irrer irrer at umich.edu (734) 647-4409
University of Michigan Hospital Radiation Oncology
519 W. William St. Ann Arbor, MI 48103
---------- Forwarded message ----------
From: Brett Cameron <brett.r.cameron at gmail.com>
Date: Fri, Jun 25, 2010 at 9:57 PM
Subject: Re: [rabbitmq-discuss] RabbitMQ C library function
amqp_simple_wait_frame takes 400 ms
To: David Wragg <david at rabbitmq.com>, Jim Irrer <irrer at umich.edu>
Cc: rabbitmq-discuss <rabbitmq-discuss at lists.rabbitmq.com>
Jim,
I believe that you are looking at a TCP/IP issue here (which you may or may
not be able to address by modifying the libRabbitMQ code). My guess is that
if you set the TCP/IP kernel parameter tcpnodelack (or whatever it is called
on your operating system) to 1 (i.e. don't delay acknowledging TCP data),
you will see things improve rather significantly. Depending on what platform
you're using, you may be able to stick a setsockopt() call (using the option
TCP_NODELACK) in amqp_socket.c between the socket() call and the connect()
call instead of having to make the chnage globally by messing with the
kernel parameter.
For what it's worth, I encountered this problem with libRabbitMQ-C on
OpenVMS just last week.. Luky for me I've seen the problem before. Seem to
recall that you guys at umich.edu used to have some OpenVMS systems...
Regards,
Brett
On Sat, Jun 26, 2010 at 11:54 AM, David Wragg <david at rabbitmq.com> wrote:
> Hi Jim,
>
> Jim Irrer <irrer at umich.edu> writes:
> > I'm working on two functions that act as a client-server pair. They
> > use two amq.direct queues to communicate. When ever either of
> > them calls the amqp_simple_wait_frame function, it does not return
> > for 436618 microseconds.
> >
> > Some other background info that might be relevant:
> >
> > If I only send messages in one direction it's really fast.
> >
> > Both processes are using separate connectors and different sockets.
> >
> > I used the amqp_consumer.c amqp_producer.c code in
> > the examples directory as a reference.
> >
> > Is there a way to avoid this delay?
>
> I'm not sure what you are really asking here. As its name suggests,
> amqp_simple_wait_frame waits for a frame to arrive. It will typically
> attempt to read from the socket connected to the AMQP server. If no
> data is available, it will block until data is available. The resulting
> delays are thus an intrinsic feature of amqp_simple_wait_frame.
>
> Are you sure that the 400ms delay does not simply reflect the wait for a
> message to arrive?
>
> I'm guessing, but perhaps the problem is that you want a single
> application to publish and consume messages concurrently, and you are
> finding that the synchronous nature of amqp_simple_wait_frame is an
> obstacle? If so, the simplest work around would be to have two threads,
> one to publish and one to consume, and open a separate AMQP connection
> in each thread.
>
> > Also ...
> >
> > Could I use the same socket in each program as long as it was only
> > used by one thread at a time?
> >
> > Could I use the same connection in each program if it was only
> > used by one thread at a time?
>
> What's the distinction between socket and connection here?
>
> librabbitmq does not do anything to explicitly support multithreading,
> but neither does it do anything to conflict with it. If you, the
> application programmer, ensure that for a given connection, only one
> thread uses librabbitmq at a time, you should be safe.
>
> --
> David Wragg
> Staff Engineer, RabbitMQ
> SpringSource, a division of VMware
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100630/93db1b08/attachment-0001.htm>
More information about the rabbitmq-discuss
mailing list