[rabbitmq-discuss] Why is RabbitMQ not persisting messages on a durable queue?

[Alexandru Scvorţov]

> I was running version 2.2.0. Upgrading to 2.3.1 did the trick. Thanks!

Happy to hear that.

> Out of curiosity what was the bug that caused persistence to fail?

Queues use a timer to force periodic fsync()s.  That timer would
sometimes *not* get restarted leading to long delays before the messages
were actually written to disk.

Cheers,
Alex

On Sat, Feb 19, 2011 at 01:19:43PM -0800, hekevintran wrote:
> I was running version 2.2.0. Upgrading to 2.3.1 did the trick. Thanks!
> 
> Out of curiosity what was the bug that caused persistence to fail?
> 
> On Feb 18, 2:55 pm, Alexandru Scvortov <alexan... at rabbitmq.com> wrote:
> > Also, are you running the latest version of the broker?  There was a bug
> > in versions of Rabbit before 2.3.0 that made the above problem worse.
> > RabbitMQ 2.3.1 can still lose messages if it's killed quickly enough,
> > but it should happen less often.
> >
> > Cheers,
> > Alex
> >
> >
> >
> >
> >
> > On Sat, Feb 19, 2011 at 12:44:17AM +0200, Alexandru Scvorţov wrote:
> > > Hi,
> >
> > > A persistent message is one Rabbit will try to write to disk, but if the
> > > server is killed before it gets the chance to do this, the message will
> > > be lost.
> >
> > > There are a few ways to remedy this, either by using transactions or by
> > > using confirms (see our recent blog posts).  That said, I don't know if
> > > you can get Celery to use any of these mechanisms.
> >
> > > Cheers,
> > > Alex
> >
> > > On Fri, Feb 18, 2011 at 01:49:46PM -0800, hekevintran wrote:
> > > > It looks like the delivery mode is already set to 2:
> >
> > > >     In [1]: from apps.test_app.tasks import add
> >
> > > >     In [2]: add.delivery_mode
> > > >     Out[2]: 2
> >
> > > > This default cannot be changed in celery as far as I know.
> >
> > > > On Feb 18, 1:40 pm, "Gavin M. Roy" <g... at myyearbook.com> wrote:
> > > > > Not sure of the celery underpinnings for specifying persistent messages,
> > > > > perhaps it's documented, but you need to set the message delivery_mode = 2
> > > > > in the message properties when calling Basic.Publish.
> >
> > > > > On Fri, Feb 18, 2011 at 4:19 PM, hekevintran <hekevint... at gmail.com> wrote:
> > > > > > I am using RabbitMQ with Django through Celery. I am using the most
> > > > > > basic setup:
> >
> > > > > >    # RabbitMQ connection settings
> > > > > >    BROKER_HOST = 'localhost'
> > > > > >    BROKER_PORT = '5672'
> > > > > >    BROKER_USER = 'guest'
> > > > > >    BROKER_PASSWORD = 'guest'
> > > > > >    BROKER_VHOST = '/'
> >
> > > > > > I imported a Celery task and queued it to run one year later. From the
> > > > > > iPython shell:
> >
> > > > > >    In [1]: from apps.test_app.tasks import add
> >
> > > > > >    In [2]: dt=datetime.datetime(2012, 2, 18, 10, 00)
> >
> > > > > >    In [3]: add.apply_async((10, 6), eta=dt)
> > > > > >    DEBUG:amqplib:Start from server, version: 8.0, properties:
> > > > > > {u'information': 'Licensed under the MPL.  Seehttp://www.rabbitmq.com/',
> > > > > > u'product': 'RabbitMQ', u'version': '2.2.0', u'copyright': 'Copyright
> > > > > > (C) 2007-2010 LShift Ltd., Cohesive Financial Technologies LLC., and
> > > > > > Rabbit Technologies Ltd.', u'platform': 'Erlang/OTP'}, mechanisms:
> > > > > > ['PLAIN', 'AMQPLAIN'], locales: ['en_US']
> > > > > >    DEBUG:amqplib:Open OK! known_hosts []
> > > > > >    DEBUG:amqplib:using channel_id: 1
> > > > > >    DEBUG:amqplib:Channel open
> > > > > >    DEBUG:amqplib:Closed channel #1
> > > > > >    Out[3]: <AsyncResult: cfc507a1-175f-438e-acea-8c989a120ab3>
> >
> > > > > > RabbitMQ received this message in the celery queue:
> >
> > > > > >    $  rabbitmqctl list_queues name messages durable
> > > > > >    Listing queues ...
> > > > > >    KTMacBook.local.celeryd.pidbox      0       false
> > > > > >    celery      1       true
> > > > > >    celeryctl_KTMacBook.local   0       true
> > > > > >    ...done.
> >
> > > > > > I then killed RabbitMQ by hitting control-C followed by 'a' to abort.
> > > > > > When I start the server again and check it with rabbitmqctl, it says
> > > > > > that there are no messages in the celery queue:
> >
> > > > > >    $  rabbitmqctl list_queues name messages durable
> > > > > >    Listing queues ...
> > > > > >    celery      0       true
> > > > > >    celeryctl_KTMacBook.local   0       true
> > > > > >    ...done.
> >
> > > > > > The celery queue was durable. Why were the messages not persisted?
> > > > > > What do I need to do to make the messages persistent?
> > > > > > _______________________________________________
> > > > > > rabbitmq-discuss mailing list
> > > > > > rabbitmq-disc... at lists.rabbitmq.com
> > > > > >https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >
> > > > > _______________________________________________
> > > > > rabbitmq-discuss mailing list
> > > > > rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > > _______________________________________________
> > > > rabbitmq-discuss mailing list
> > > > rabbitmq-disc... at lists.rabbitmq.com
> > > >https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > _______________________________________________
> > > rabbitmq-discuss mailing list
> > > rabbitmq-disc... at lists.rabbitmq.com
> > >https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss