What I'm trying to implement is a simple client-server paradigm where each<br>is single threaded. The client sends a message to the server, and then<br>waits for a reply by posting a read on a separate reply queue. When the<br>
server gets the message, it processes it, builds a reply, and sends the<br>reply to the client.<br><br>I've re-built the rabbitmq-c-default library and inserted print statements<br>that show the time in microseconds, and the series of events is a loop that<br>
basically does the following:<br><br><div style="margin-left: 40px;">server waits on read<br><br>client sends message and then waits for reply<br><br>server gets message<br><br>server performs trivial processing<br><br>server sends reply<br>
<br>client gets reply<br></div><br>Below is a listing of events with timestamps in microseconds. The initial 3172304<br>microsecond delay is the time it took me to start the client in a different window<br>after starting the server. When a process is "waiting on read", it is actually the<br>
system read call that is posted on the socket. I put the print statement inside the<br>AMQP library call.<br><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">relative elapsed</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> time time operation</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">-------- ------- ---------------------------------------</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 0 0 server waiting on read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 3172304 3172304 client calling amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 3172331 27 client returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 3172337 6 client waiting on read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 3383774 211437 server returned from read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 3383892 118 server 1 message: 1</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 3383898 6 server calling amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 3383927 29 server returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 3383938 11 server waiting on read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 3497244 113306 client returned from read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 3497279 35 client message: 1</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 3497288 9 client calling amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 3497310 22 client returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 3497315 5 client waiting on read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 3715343 218028 server returned from read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 3715374 31 server 2 message: 2</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 3715378 4 server calling amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 3715400 22 server returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 3715405 5 server waiting on read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 3934206 218801 client returned from read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 3934236 30 client message: 2</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 3934243 7 client calling amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 3934265 22 client returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 3934270 5 client waiting on read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 4153113 218843 server returned from read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 4153151 38 server 3 message: 3</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 4153158 7 server calling amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 4153189 31 server returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 4153197 8 server waiting on read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 4371982 218785 client returned from read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 4372017 35 client message: 3</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 4372027 10 client calling amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 4372091 64 client returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 4372099 8 client waiting on read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 4590627 218528 server returned from read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 4590657 30 server 4 message: 4</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 4590661 4 server calling amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 4590682 21 server returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 4590687 5 server waiting on read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 4809133 218446 client returned from read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 4809162 29 client message: 4</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 4809169 7 client calling amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 4809190 21 client returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 4809195 5 client waiting on read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 5028276 219081 server returned from read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 5028308 32 server 5 message: 5</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 5028312 4 server calling amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 5028334 22 server returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> 5030059 1725 client returned from read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> 5030077 18 client message: 5</span><br style="font-family: courier new,monospace;">
<br>Each side seems to be introducing a delay that is suspiciously close to 200<br>milliseconds, that may be some system configured value.<br><br>I tried TCP_NODELAY (it looks like TCP_NODELACK is <span style="font-size: 11pt;">only </span>in AIX)<br>
but it had no effect. Setting this with setsockopt turns off Nagle's<br>algorithm which is set by default and tries to optimize socket performance<br>for large data transfers by deferring sends until either there is enough<br>
data to send a full packet or a timeout occurs. Oddly, I had to be root<br>to set this option.<br><br>If you have any idea as to why the extra delay is being introduced, please<br>let me know (and especially how to fix it :) )<br>
<br>Thanks for any insights,<br><br>- Jim<br><br>Jim Irrer <a href="mailto:irrer@umich.edu">irrer@umich.edu</a> (734) 647-4409<br>University of Michigan Hospital Radiation Oncology<br>519 W. William St. Ann Arbor, MI 48103<br>
<br><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Brett Cameron</b> <span dir="ltr"><<a href="mailto:brett.r.cameron@gmail.com">brett.r.cameron@gmail.com</a>></span><br>
Date: Fri, Jun 25, 2010 at 9:57 PM<br>Subject: Re: [rabbitmq-discuss] RabbitMQ C library function amqp_simple_wait_frame takes 400 ms<br>To: David Wragg <<a href="mailto:david@rabbitmq.com">david@rabbitmq.com</a>>, Jim Irrer <<a href="mailto:irrer@umich.edu">irrer@umich.edu</a>><br>
Cc: rabbitmq-discuss <<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a>><br><br><br><div>Jim,</div>
<div> </div>
<div>I believe that you are looking at a TCP/IP issue here (which you may or may not be able to address by modifying the libRabbitMQ code). My guess is that if you set the TCP/IP kernel parameter tcpnodelack (or whatever it is called on your operating system) to 1 (i.e. don't delay acknowledging TCP data), you will see things improve rather significantly. Depending on what platform you're using, you may be able to stick a setsockopt() call (using the option <span style="font-size: 11pt;">TCP_NODELACK) in amqp_socket.c between the socket() call and the connect() call instead of having to make the chnage globally by messing with the kernel parameter.</span><br>
</div>
<div>For what it's worth, I encountered this problem with libRabbitMQ-C on OpenVMS just last week.. Luky for me I've seen the problem before. Seem to recall that you guys at <a href="http://umich.edu" target="_blank">umich.edu</a> used to have some OpenVMS systems...</div>
<div> </div>
<div>Regards,</div>
<div>Brett</div><font color="#888888">
<div><br><br> </div>
</font><div class="gmail_quote"><div><div></div><div class="h5">On Sat, Jun 26, 2010 at 11:54 AM, David Wragg <span dir="ltr"><<a href="mailto:david@rabbitmq.com" target="_blank">david@rabbitmq.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="padding-left: 1ex; margin: 0px 0px 0px 0.8ex; border-left: 1px solid rgb(204, 204, 204);"><div><div></div><div class="h5">Hi Jim,<br>
<div><br>Jim Irrer <<a href="mailto:irrer@umich.edu" target="_blank">irrer@umich.edu</a>> writes:<br>> I'm working on two functions that act as a client-server pair. They<br>> use two amq.direct queues to communicate. When ever either of<br>
> them calls the amqp_simple_wait_frame function, it does not return<br>> for 436618 microseconds.<br>><br>> Some other background info that might be relevant:<br>><br>> If I only send messages in one direction it's really fast.<br>
><br>> Both processes are using separate connectors and different sockets.<br>><br>> I used the amqp_consumer.c amqp_producer.c code in<br>> the examples directory as a reference.<br>><br>> Is there a way to avoid this delay?<br>
<br></div>I'm not sure what you are really asking here. As its name suggests,<br>amqp_simple_wait_frame waits for a frame to arrive. It will typically<br>attempt to read from the socket connected to the AMQP server. If no<br>
data is available, it will block until data is available. The resulting<br>delays are thus an intrinsic feature of amqp_simple_wait_frame.<br><br>Are you sure that the 400ms delay does not simply reflect the wait for a<br>
message to arrive?<br><br>I'm guessing, but perhaps the problem is that you want a single<br>application to publish and consume messages concurrently, and you are<br>finding that the synchronous nature of amqp_simple_wait_frame is an<br>
obstacle? If so, the simplest work around would be to have two threads,<br>one to publish and one to consume, and open a separate AMQP connection<br>in each thread.<br>
<div><br>> Also ...<br>><br>> Could I use the same socket in each program as long as it was only<br>> used by one thread at a time?<br>><br>> Could I use the same connection in each program if it was only<br>
> used by one thread at a time?<br><br></div>What's the distinction between socket and connection here?<br><br>librabbitmq does not do anything to explicitly support multithreading,<br>but neither does it do anything to conflict with it. If you, the<br>
application programmer, ensure that for a given connection, only one<br>thread uses librabbitmq at a time, you should be safe.<br></div></div><font color="#888888"><div><div></div><div class="h5"><br>--<br>David Wragg<br>
Staff Engineer, RabbitMQ<br>SpringSource, a division of VMware<br></div></div><div class="im">
_______________________________________________<br>rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></font></blockquote></div><br>
</div><br>