[rabbitmq-discuss] Pika - how to properly publish from a periodic background thread?

[Aaron Voelker]

Thanks for the reply Marek.

I think you're right that having two separate connections in separate
threads would work.

But I think I found a more elegant solution that avoids threading
altogether, by using an IOLoop. Basically, I used the Tornado adapter
for Pika: http://pika.github.com/connecting.html#adapters-tornado-connection-tornadoconnection

Then I was able to leverage Tornado's PeriodicCallback and add that to
the same IOLoop to do background publishing.
https://github.com/facebook/tornado/blob/master/tornado/ioloop.py

Everything is now singly threaded, and it worked like a charm. Hoping
someone will find this info helpful.

Aaron

On Sep 26, 10:00 am, Marek Majkowski <maje... at gmail.com> wrote:
> On Thu, Sep 22, 2011 at 21:26, Aaron Voelker <aa... at contextlogic.com> wrote:
> > I'm using Python 2.6.6, Ubuntu 10.10, and Pika 0.9.5.
>
> > I have a situation where I want to publish to a queue from a
> > background thread at a periodic interval. The main thread is consuming
> > on some different callback.
>
> > I assumed (incorrectly) that wrapping my callback and publish in a
> > mutex would solve this problem. I ran into some weird behaviour before
> > I realized that maybe this isn't the correct approach.
>
> > What happens is that the worker I publish to from the background
> > thread will send work back to the callback for the main thread.
> > Normally, this is fine, but in weird situations (race condition?),
> > after a few hours of running my application, it will get Pika into an
> > unrecoverable state. Some sample code which duplicates this problem,
> > along with the resulting error messages, is included in this message
> > in case someone happens to Google for this error message (which got me
> > next to nowhere).
>
> > What is the correct way to do this? Note that both the callback and
> > the scheduler have to have shared access to the same variables (hence
> > the mutex), and performance does NOT matter (the scheduler runs once
> > per minute, and doesn't do too much.
>
> What about just opening two separate connections, one in each thread?
>
> I'm not sure how "reactors" work in Pika, but I guess there should be a
> way to have two separate event loops.
>
> Cheers,
>   Marek
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss