What I'm trying to implement is a simple client-server paradigm where each<br>is single threaded.� The client sends a message to the server, and then<br>waits for a reply by posting a read on a separate reply queue.� When the<br>
server gets the message, it processes it, builds a reply, and sends the<br>reply to the client.<br><br>I've re-built the rabbitmq-c-default library and inserted print statements<br>that show the time in microseconds, and the series of events is a loop that<br>
basically does the following:<br><br><div style="margin-left: 40px;">server waits on read<br><br>client sends message and then waits for reply<br><br>server gets message<br><br>server performs trivial processing<br><br>server sends reply<br>
<br>client gets reply<br></div><br>Below is a listing of events with timestamps in microseconds.� The initial 3172304<br>microsecond delay is the time it took me to start the client in a different window<br>after starting the server.� When a process is "waiting on read", it is actually the<br>
system read call that is posted on the socket.� I put the print statement inside the<br>AMQP library call.<br><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">relative�� elapsed</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">� time����� time��� operation</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">--------�� -------� ---------------------------------------</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">������ 0�������� 0� server waiting on read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�3172304�� 3172304� client calling amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�3172331������� 27� client returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�3172337�������� 6� client waiting on read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�3383774��� 211437� server returned from read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�3383892������ 118� server 1 message: 1</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�3383898�������� 6� server calling amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�3383927������� 29� server returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�3383938������� 11� server waiting on read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�3497244��� 113306� client returned from read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�3497279������� 35� client message: 1</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�3497288�������� 9� client calling amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�3497310������� 22� client returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�3497315�������� 5� client waiting on read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�3715343��� 218028� server returned from read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�3715374������� 31� server 2 message: 2</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�3715378�������� 4� server calling amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�3715400������� 22� server returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�3715405�������� 5� server waiting on read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�3934206��� 218801� client returned from read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�3934236������� 30� client message: 2</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�3934243�������� 7� client calling amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�3934265������� 22� client returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�3934270�������� 5� client waiting on read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�4153113��� 218843� server returned from read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�4153151������� 38� server 3 message: 3</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�4153158�������� 7� server calling amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�4153189������� 31� server returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�4153197�������� 8� server waiting on read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�4371982��� 218785� client returned from read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�4372017������� 35� client message: 3</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�4372027������� 10� client calling amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�4372091������� 64� client returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�4372099�������� 8� client waiting on read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�4590627��� 218528� server returned from read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�4590657������� 30� server 4 message: 4</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�4590661�������� 4� server calling amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�4590682������� 21� server returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�4590687�������� 5� server waiting on read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�4809133��� 218446� client returned from read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�4809162������� 29� client message: 4</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�4809169�������� 7� client calling amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�4809190������� 21� client returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�4809195�������� 5� client waiting on read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�5028276��� 219081� server returned from read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�5028308������� 32� server 5 message: 5</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�5028312�������� 4� server calling amqp_basic_publish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�5028334������� 22� server returned from amqp_basic_publish</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">�5030059����� 1725� client returned from read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">�5030077������� 18� client message: 5</span><br style="font-family: courier new,monospace;">
<br>Each side seems to be introducing a delay that is suspiciously close to 200<br>milliseconds, that may be some system configured value.<br><br>I tried TCP_NODELAY (it looks like TCP_NODELACK is <span style="font-size: 11pt;">only </span>in AIX)<br>
but it had no effect.� Setting this with setsockopt turns off Nagle's<br>algorithm which is set by default and tries to optimize socket performance<br>for large data transfers by deferring sends until either there is enough<br>
data to send a full packet or a timeout occurs.� Oddly, I had to be root<br>to set this option.<br><br>If you have any idea as to why the extra delay is being introduced, please<br>let me know (and especially how to fix it :)� )<br>
<br>Thanks for any insights,<br><br>- Jim<br><br>Jim Irrer � � <a href="mailto:irrer@umich.edu">irrer@umich.edu</a> � � � (734) 647-4409<br>University of Michigan Hospital Radiation Oncology<br>519 W. William St. � � � � � � Ann Arbor, MI 48103<br>
<br><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Brett Cameron</b> <span dir="ltr"><<a href="mailto:brett.r.cameron@gmail.com">brett.r.cameron@gmail.com</a>></span><br>
Date: Fri, Jun 25, 2010 at 9:57 PM<br>Subject: Re: [rabbitmq-discuss] RabbitMQ C library function amqp_simple_wait_frame takes 400 ms<br>To: David Wragg <<a href="mailto:david@rabbitmq.com">david@rabbitmq.com</a>>, Jim Irrer <<a href="mailto:irrer@umich.edu">irrer@umich.edu</a>><br>
Cc: rabbitmq-discuss <<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a>><br><br><br><div>Jim,</div>
<div>�</div>
<div>I believe that you are looking at a TCP/IP issue here (which you may or may not be able to address by modifying the libRabbitMQ code).�My guess is that if you set the TCP/IP kernel parameter tcpnodelack�(or whatever it is called on your operating system) to 1 (i.e. don't delay acknowledging TCP data), you will see things improve rather significantly. Depending on what platform you're using, you may be able to stick a setsockopt() call (using the option <span style="font-size: 11pt;">TCP_NODELACK) in amqp_socket.c between the socket() call and the connect() call instead of having to make the chnage globally by messing with the kernel parameter.</span><br>
</div>
<div>For what it's worth, I encountered this problem with libRabbitMQ-C on OpenVMS just last week.. Luky for me I've seen the problem before. Seem to recall that you guys at <a href="http://umich.edu" target="_blank">umich.edu</a> used to have some OpenVMS systems...</div>
<div>�</div>
<div>Regards,</div>
<div>Brett</div><font color="#888888">
<div><br><br>�</div>
</font><div class="gmail_quote"><div><div></div><div class="h5">On Sat, Jun 26, 2010 at 11:54 AM, David Wragg <span dir="ltr"><<a href="mailto:david@rabbitmq.com" target="_blank">david@rabbitmq.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="padding-left: 1ex; margin: 0px 0px 0px 0.8ex; border-left: 1px solid rgb(204, 204, 204);"><div><div></div><div class="h5">Hi Jim,<br>
<div><br>Jim Irrer <<a href="mailto:irrer@umich.edu" target="_blank">irrer@umich.edu</a>> writes:<br>> I'm working on two functions that act as a client-server pair. �They<br>> use two amq.direct queues to communicate. �When ever either of<br>
> them calls the amqp_simple_wait_frame function, it does not return<br>> for 436618 microseconds.<br>><br>> Some other background info that might be relevant:<br>><br>> If I only send messages in one direction it's really fast.<br>
><br>> Both processes are using separate connectors and different sockets.<br>><br>> I used the amqp_consumer.c amqp_producer.c code in<br>> the examples directory as a reference.<br>><br>> Is there a way to avoid this delay?<br>
<br></div>I'm not sure what you are really asking here. �As its name suggests,<br>amqp_simple_wait_frame waits for a frame to arrive. �It will typically<br>attempt to read from the socket connected to the AMQP server. �If no<br>
data is available, it will block until data is available. �The resulting<br>delays are thus an intrinsic feature of amqp_simple_wait_frame.<br><br>Are you sure that the 400ms delay does not simply reflect the wait for a<br>
message to arrive?<br><br>I'm guessing, but perhaps the problem is that you want a single<br>application to publish and consume messages concurrently, and you are<br>finding that the synchronous nature of amqp_simple_wait_frame is an<br>
obstacle? �If so, the simplest work around would be to have two threads,<br>one to publish and one to consume, and open a separate AMQP connection<br>in each thread.<br>
<div><br>> Also ...<br>><br>> Could I use the same socket in each program as long as it was only<br>> used by one thread at a time?<br>><br>> Could I use the same connection in each program if it was only<br>
> used by one thread at a time?<br><br></div>What's the distinction between socket and connection here?<br><br>librabbitmq does not do anything to explicitly support multithreading,<br>but neither does it do anything to conflict with it. �If you, the<br>
application programmer, ensure that for a given connection, only one<br>thread uses librabbitmq at a time, you should be safe.<br></div></div><font color="#888888"><div><div></div><div class="h5"><br>--<br>David Wragg<br>
Staff Engineer, RabbitMQ<br>SpringSource, a division of VMware<br></div></div><div class="im">
_______________________________________________<br>rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></font></blockquote></div><br>
</div><br>