[rabbitmq-discuss] Performance degrades with increasing queue depth

[Suhail Doshi]

This is the gist of mine:
def get_channel(self):
        """
            Creates a persistent, ack based, non-exclusive queue/exchange
using the routing key event

            Non-exclusivity allows any consumer to grab from the queue
        """
        channel = self.connection.channel()

        channel.queue_declare(queue='storage', durable=True,
exclusive=False, auto_delete=False)
        channel.exchange_declare(exchange='records', type='direct',
durable=True, auto_delete=False)
        channel.queue_bind(queue='storage', exchange='records',
routing_key='event')

        channel.basic_consume(
            queue='storage', no_ack=False, callback=self.recv_event_record,
            consumer_tag=self.consumer_tag
        )

        self.channel = channel

        return channel

    def recv_event_record(self, data):
        """
            Cleans up the record, passes it on for processing
        """

        item = cjson.decode(unicode(data.body).encode("utf8"))

        [processing...]

        logging.info('[id: ' + str(self.consumer_id) + '] Received queue
item, processing...')

        self.process_record(record)
        self.channel.basic_ack(data.delivery_tag)

        logging.info('[id: ' + str(self.consumer_id) + '] Processing
complete, acknowledged.')

    def process_record(self, item):
        process(item)

def connect_consumer():
    channel = get_channel()

    logging.info('Consumer started')

    while True:
        channel.wait()

On Tue, Sep 8, 2009 at 7:01 AM, Chuck Remes <cremes.devlist at mac.com> wrote:

> Aisha is mistaken. The ruby amqp library does not use libevent
> anywhere. It uses EventMachine under the covers which is a home-grown
> reactor written in C that uses select, poll and kqueue depending on
> the platform.
>
> cr
>
>
> On Sep 8, 2009, at 8:33 AM, Ben Hood wrote:
>
> > Aisha,
> >
> > Not wanting to hijack this thread another time, but....
> >
> > On Mon, Sep 7, 2009 at 10:44 PM, Aisha
> > Fenton<aisha.fenton at gmail.com> wrote:
> >> The library I'm using (http://github.com/celldee/bunny/tree/master)
> >> is
> >> synchronous only. I'm considering switching to (
> http://github.com/tmm1/amqp/tree/master
> >> ) which is async and uses libevent. However, I've read that there are
> >> problems with msg acknowledges in apps that use libevent, because the
> >> OS can buffer the ack responses. And although I'm not using ack
> >> yet, I
> >> want to in my production system.
> >
> > I am interested as to what the specific issue is with sending acks
> > with libevent - are you saying that the app can issue an ack and this
> > can potentially go AWOL between the the libevent API returning and the
> > point it time when the AMQP frame is actually written to the socket?
> >
> > Ben
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>



-- 
http://mixpanel.com
Blog: http://blog.mixpanel.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090908/5e7a5b4f/attachment.htm