[rabbitmq-discuss] Pika - how to properly publish from a periodic background thread?

[Marek Majkowski]

On Tue, Sep 27, 2011 at 03:14, Aaron Voelker <aaron at contextlogic.com> wrote:
> Thanks for the reply Marek.
>
> I think you're right that having two separate connections in separate
> threads would work.
>
> But I think I found a more elegant solution that avoids threading
> altogether, by using an IOLoop. Basically, I used the Tornado adapter
> for Pika: http://pika.github.com/connecting.html#adapters-tornado-connection-tornadoconnection
>
> Then I was able to leverage Tornado's PeriodicCallback and add that to
> the same IOLoop to do background publishing.
> https://github.com/facebook/tornado/blob/master/tornado/ioloop.py
>
> Everything is now singly threaded, and it worked like a charm. Hoping
> someone will find this info helpful.

I'm glad you got that working. Avoiding threads is almost always a
better solution.

Marek

> On Sep 26, 10:00 am, Marek Majkowski <maje... at gmail.com> wrote:
>> On Thu, Sep 22, 2011 at 21:26, Aaron Voelker <aa... at contextlogic.com> wrote:
>> > I'm using Python 2.6.6, Ubuntu 10.10, and Pika 0.9.5.
>>
>> > I have a situation where I want to publish to a queue from a
>> > background thread at a periodic interval. The main thread is consuming
>> > on some different callback.
>>
>> > I assumed (incorrectly) that wrapping my callback and publish in a
>> > mutex would solve this problem. I ran into some weird behaviour before
>> > I realized that maybe this isn't the correct approach.
>>
>> > What happens is that the worker I publish to from the background
>> > thread will send work back to the callback for the main thread.
>> > Normally, this is fine, but in weird situations (race condition?),
>> > after a few hours of running my application, it will get Pika into an
>> > unrecoverable state. Some sample code which duplicates this problem,
>> > along with the resulting error messages, is included in this message
>> > in case someone happens to Google for this error message (which got me
>> > next to nowhere).
>>
>> > What is the correct way to do this? Note that both the callback and
>> > the scheduler have to have shared access to the same variables (hence
>> > the mutex), and performance does NOT matter (the scheduler runs once
>> > per minute, and doesn't do too much.
>>
>> What about just opening two separate connections, one in each thread?
>>
>> I'm not sure how "reactors" work in Pika, but I guess there should be a
>> way to have two separate event loops.
>>
>> Cheers,
>>   Marek
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>