<div dir="ltr">You rock Simon! So much great information. I found my problem and it was indeed a gen_server timeout due to a missed return in my confirmation chain going back up the chain to the rest interface where I respond to the client that posted the message for publishing.<div>
<br></div><div>However, that said, I am curious about the option of just using <span style="font-size:13px;font-family:arial,sans-serif">amqp_channel:wait_for_</span><u style="font-size:13px;font-family:arial,sans-serif"></u><span style="font-size:13px;font-family:arial,sans-serif">confirms</span><span style="font-size:13px;font-family:arial,sans-serif">/1. I did not use this option for a couple of reasons. </span></div>
<div><span style="font-size:13px;font-family:arial,sans-serif"><br></span></div><div><span style="font-size:13px;font-family:arial,sans-serif">First was the fact that using the registered handler is mentioned quite often as the preferred option. My understanding was that using the </span><span style="font-size:13px;font-family:arial,sans-serif">amqp_channel:wait_for_</span><u style="font-size:13px;font-family:arial,sans-serif"></u><font face="arial, sans-serif">confirms would cause the channel to wait for a response before another message could be published on that channel. If this is true, it seems like it could slow-down publishing more than registering a handler and letting publishing continue. Is this correct? Or am I misunderstanding the whole workflow?</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">Speaking of misunderstandings; the second reason I did not use wait_for_confirms was that I didn't completely understand the definition:</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">"</font><span style="color:rgb(0,0,0);font-family:Times;font-size:medium">Wait until all messages published since the last call have been either ack'd or nack'd by the broker; or until timeout elapses."</span></div>
<div><span style="color:rgb(0,0,0);font-family:Times;font-size:medium"><br></span></div>What exactly does it mean by the "last call"? This sounded to me as if it may get in a situation that if calls are coming-in rapidly enough that they are publishing new messages prior to the previous message getting ack'd, it would not respond until all such queued messages were ack'd/nack'd. I realized this was unlikely, but could not come-up with documentation clear enough to convince me otherwise and this would not work for me as I need to respond to each publish individually.<div>
<br></div><div>Thanks again for the, characteristically, excellent and comprehensive reponse.</div><div><br></div><div>Best,</div><div><br></div><div>Ryan</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Mon, Apr 7, 2014 at 4:20 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
It's going to be hard to debug this without seeing your code, especially since it sounds like it is quite likely to be a general Erlang problem.<br>
<br>
Having said that, here are some general pointers:<br>
<br>
On 04/04/2014 20:44, Ryan Brown wrote:<br>
<snip><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Then on publish, I select which one to use based<br>
on the values passed-down the stack from message intake. Set self() as<br>
the confirm handler and respond with a wait so that up the stack I can<br>
initiate a receive loop in the message intake which is not a gen_server.<br>
</blockquote>
<br>
Have you seen amqp_channel:wait_for_<u></u>confirms/1? That might be easier than handling the confirms yourself if all you are going to do is wait for them anyway.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I have debug messages throughout and it appears that everything is<br>
happening as expected. I trace the message through the publish path. To<br>
being published and passing the wait back up to the receiver. I also see<br>
where the basic.ack is received and the publish_confirm acknowledgement<br>
passed all the way up to the receive loop. However, then things go awry.<br>
I get the attached dump.<br>
</blockquote>
<br>
It's very hard to see what is going wrong here in detail, but the surface issue is that your gen_server took more than 5 seconds to respond to a request - note that gen_server:call/2 will time out after 5 seconds.<br>
<br>
For this reason we never use gen_server:call/2 (we have a CI task that explodes if we accidentally commit code using it!); we always use call/3 with a timeout of 'infinity'. IMO this is a misfeature in Erlang.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
My take-away, although likely I am wrong, is that somewhere, apparently<br>
in amqp_balanced_publisher, a gen_server call is timing-out while<br>
waiting for a response. But, I seem to be able to trace the responses<br>
through the whole stack so I am thoroughly confused.<br>
</blockquote>
<br>
That's correct. Of course, it should be quite rare / require heavy load for the 5s timeout to fail because the operation simply took too long. It's quite possible that the gen_server you are calling into is deadlocked or stuck in some other way.<br>
<br>
I would debug this as follows:<br>
<br>
1) Change the call/2 to call/3 with infinity.<br>
2) If the program now hangs rather than times out, launch 'observer' (<a href="http://www.erlang.org/doc/apps/observer/observer_ug.html" target="_blank">http://www.erlang.org/doc/<u></u>apps/observer/observer_ug.html</a><u></u>), go and find the server you are calling into, and see what it's current stacktrace is. You may wish to give this process a registered name temporarily so you can find it more easily.<br>
<br>
Cheers, Simon<br>
<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>-rb
</div>