<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Ok, I think we *might* be out of the woods. Earlier in the day Alexis & Co. told us that publishing on the channel might not be thread safe in our application. We made a change earlier in the day, but it looks like it never really made it to production. Verified this evening about an hour ago and now things are running better w/ the sync block around on the publish calls. We think that the frames were getting clobbered and as a result killing the connection and giving us socket closed exceptions. We saw this at one point while running the trace:<div><br class="webkit-block-placeholder"></div><div><span class="Apple-style-span" style="font-family: Arial; font-size: 10px; "> Exception in thread "Thread-4" com.rabbitmq.client.UnexpectedFrameError<br> at com.rabbitmq.client.impl.AMQCommand$Assembler.handleFrame(AMQCommand.java:291)<br> at com.rabbitmq.tools.Tracer$DirectionHandler.doFrame(Tracer.java:158)<br> at com.rabbitmq.tools.Tracer$DirectionHandler.run(Tracer.java:170)<br> at java.lang.Thread.run(Thread.java:595)</span><br><div><br class="webkit-block-placeholder"></div><div>This was our biggest traffic day in history - a ton of which is coming from AOL. In the last hour or so we've pushed about 48k requests through a single broker and the payload is about 5k each.</div><div><br class="webkit-block-placeholder"></div><div>That said, things are looking better. Now we have sluggish publish times because everyone's blocked on the publish. However, I think we can just create more channels and spread the load that way. We also have a subset of export requests that should get processed first before others, so we'll be leveraging the priority level in the protocol. Real excited about that.</div><div><br></div><div>Many thanks to the rabbitMQ team. You guys have a responsiveness that's unheard of! </div><div><br class="webkit-block-placeholder"></div><div>-matt</div><div><div> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div><font color="gray" face="Arial" size="2"><span><br class="Apple-interchange-newline">--</span></font><br><a href="http://www.meez.com/"><img src="http://images.meez.com/user05/04/04_10001605581.gif" width="175" height="233" alt="meez" align="left" style="border-top-style: none; border-right-style: none; border-bottom-style: none; border-left-style: none; border-width: initial; border-color: initial; "></a><font color="gray" face="Arial" size="2"><span>Matt Darling</span></font><br><font color="gray" face="Arial" size="2"><span>Director, Server Engineering</span></font></div><div><font color="gray" face="Arial" size="2"><span>Meez</span></font><br><font color="gray" face="Arial" size="2"><span>182 2nd Street</span></font><font color="gray" face="Arial" size="2"><span>, 5th Floor</span></font><br><font color="gray" face="Arial" size="2"><span>San Francisco</span></font><font color="gray" face="Arial" size="2"><span>, CA 94105</span></font><br><font color="gray" face="Arial" size="2"><span>(m) 415.505.4623</span></font><br><font color="gray" face="Arial" size="2"><span><a href="mailto:matt@corp.meez.com" target="_blank">matt@corp.meez.com</a></span></font><br></div><br></span><br class="Apple-interchange-newline"> </div><br></div></div></body></html>