[rabbitmq-discuss] Fwd: RabbitMQ C library function amqp_simple_wait_frame takes 400 ms

Wed Jun 30 21:54:55 BST 2010

What I'm trying to implement is a simple client-server paradigm where each
is single threaded.  The client sends a message to the server, and then
waits for a reply by posting a read on a separate reply queue.  When the
server gets the message, it processes it, builds a reply, and sends the
reply to the client.

I've re-built the rabbitmq-c-default library and inserted print statements
that show the time in microseconds, and the series of events is a loop that
basically does the following:

server waits on read

client sends message and then waits for reply

server gets message

server performs trivial processing

server sends reply

client gets reply

Below is a listing of events with timestamps in microseconds.  The initial
3172304
microsecond delay is the time it took me to start the client in a different
window
after starting the server.  When a process is "waiting on read", it is
actually the
system read call that is posted on the socket.  I put the print statement
inside the
AMQP library call.

relative   elapsed
  time      time    operation
--------   -------  ---------------------------------------
       0         0  server waiting on read
 3172304   3172304  client calling amqp_basic_publish
 3172331        27  client returned from amqp_basic_publish
 3172337         6  client waiting on read
 3383774    211437  server returned from read
 3383892       118  server 1 message: 1
 3383898         6  server calling amqp_basic_publish
 3383927        29  server returned from amqp_basic_publish
 3383938        11  server waiting on read
 3497244    113306  client returned from read
 3497279        35  client message: 1
 3497288         9  client calling amqp_basic_publish
 3497310        22  client returned from amqp_basic_publish
 3497315         5  client waiting on read
 3715343    218028  server returned from read
 3715374        31  server 2 message: 2
 3715378         4  server calling amqp_basic_publish
 3715400        22  server returned from amqp_basic_publish
 3715405         5  server waiting on read
 3934206    218801  client returned from read
 3934236        30  client message: 2
 3934243         7  client calling amqp_basic_publish
 3934265        22  client returned from amqp_basic_publish
 3934270         5  client waiting on read
 4153113    218843  server returned from read
 4153151        38  server 3 message: 3
 4153158         7  server calling amqp_basic_publish
 4153189        31  server returned from amqp_basic_publish
 4153197         8  server waiting on read
 4371982    218785  client returned from read
 4372017        35  client message: 3
 4372027        10  client calling amqp_basic_publish
 4372091        64  client returned from amqp_basic_publish
 4372099         8  client waiting on read
 4590627    218528  server returned from read
 4590657        30  server 4 message: 4
 4590661         4  server calling amqp_basic_publish
 4590682        21  server returned from amqp_basic_publish
 4590687         5  server waiting on read
 4809133    218446  client returned from read
 4809162        29  client message: 4
 4809169         7  client calling amqp_basic_publish
 4809190        21  client returned from amqp_basic_publish
 4809195         5  client waiting on read
 5028276    219081  server returned from read
 5028308        32  server 5 message: 5
 5028312         4  server calling amqp_basic_publish
 5028334        22  server returned from amqp_basic_publish
 5030059      1725  client returned from read
 5030077        18  client message: 5

Each side seems to be introducing a delay that is suspiciously close to 200
milliseconds, that may be some system configured value.

I tried TCP_NODELAY (it looks like TCP_NODELACK is only in AIX)
but it had no effect.  Setting this with setsockopt turns off Nagle's
algorithm which is set by default and tries to optimize socket performance
for large data transfers by deferring sends until either there is enough
data to send a full packet or a timeout occurs.  Oddly, I had to be root
to set this option.

If you have any idea as to why the extra delay is being introduced, please
let me know (and especially how to fix it :)  )

Thanks for any insights,

- Jim

Jim Irrer     irrer at umich.edu       (734) 647-4409
University of Michigan Hospital Radiation Oncology
519 W. William St.             Ann Arbor, MI 48103

---------- Forwarded message ----------
From: Brett Cameron <brett.r.cameron at gmail.com>
Date: Fri, Jun 25, 2010 at 9:57 PM
Subject: Re: [rabbitmq-discuss] RabbitMQ C library function
amqp_simple_wait_frame takes 400 ms
To: David Wragg <david at rabbitmq.com>, Jim Irrer <irrer at umich.edu>
Cc: rabbitmq-discuss <rabbitmq-discuss at lists.rabbitmq.com>

Jim,

I believe that you are looking at a TCP/IP issue here (which you may or may
not be able to address by modifying the libRabbitMQ code). My guess is that
if you set the TCP/IP kernel parameter tcpnodelack (or whatever it is called
on your operating system) to 1 (i.e. don't delay acknowledging TCP data),
you will see things improve rather significantly. Depending on what platform
you're using, you may be able to stick a setsockopt() call (using the option
TCP_NODELACK) in amqp_socket.c between the socket() call and the connect()
call instead of having to make the chnage globally by messing with the
kernel parameter.
 For what it's worth, I encountered this problem with libRabbitMQ-C on
OpenVMS just last week.. Luky for me I've seen the problem before. Seem to
recall that you guys at umich.edu used to have some OpenVMS systems...

Regards,
Brett

On Sat, Jun 26, 2010 at 11:54 AM, David Wragg <david at rabbitmq.com> wrote:

> Hi Jim,
>
> Jim Irrer <irrer at umich.edu> writes:
> > I'm working on two functions that act as a client-server pair.  They
> > use two amq.direct queues to communicate.  When ever either of
> > them calls the amqp_simple_wait_frame function, it does not return
> > for 436618 microseconds.
> >
> > Some other background info that might be relevant:
> >
> > If I only send messages in one direction it's really fast.
> >
> > Both processes are using separate connectors and different sockets.
> >
> > I used the amqp_consumer.c amqp_producer.c code in
> > the examples directory as a reference.
> >
> > Is there a way to avoid this delay?
>
> I'm not sure what you are really asking here.  As its name suggests,
> amqp_simple_wait_frame waits for a frame to arrive.  It will typically
> attempt to read from the socket connected to the AMQP server.  If no
> data is available, it will block until data is available.  The resulting
> delays are thus an intrinsic feature of amqp_simple_wait_frame.
>
> Are you sure that the 400ms delay does not simply reflect the wait for a
> message to arrive?
>
> I'm guessing, but perhaps the problem is that you want a single
> application to publish and consume messages concurrently, and you are
> finding that the synchronous nature of amqp_simple_wait_frame is an
> obstacle?  If so, the simplest work around would be to have two threads,
> one to publish and one to consume, and open a separate AMQP connection
> in each thread.
>
> > Also ...
> >
> > Could I use the same socket in each program as long as it was only
> > used by one thread at a time?
> >
> > Could I use the same connection in each program if it was only
> > used by one thread at a time?
>
> What's the distinction between socket and connection here?
>
> librabbitmq does not do anything to explicitly support multithreading,
> but neither does it do anything to conflict with it.  If you, the
> application programmer, ensure that for a given connection, only one
> thread uses librabbitmq at a time, you should be safe.
>
> --
> David Wragg
> Staff Engineer, RabbitMQ
> SpringSource, a division of VMware
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100630/93db1b08/attachment-0001.htm>