[rabbitmq-discuss] AMQPChannelException: (404, u"NOT_FOUND - no exchange '4042acd4bbec471aafc557af39ee0efe' in vhost '/'", (60, 40), 'Channel.basic_publish')

[Ask Solem]

On 18 Dec 2012, at 08:54, Alex Lyakas <alex at zadarastorage.com> wrote:

> Greetings all,
> 
> I am using rabbitmq server + kombu + pyamqplib on a stock Ubuntu-Precise. Below are the versions of the components:
> rabbitmq-server 2.7.1-0ubuntu4
> python-amqplib  1.0.0+ds-1
> python-kombu 1.4.3-1
> 
> The issue I am seeing is the “no exchange” exception, which I hit when opening a new channel on an existing connection. Here is a typical crash stack:
> TRACE: File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 159, in channel
> TRACE: chan = self.transport.create_channel(self.connection)
> TRACE: File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 235, in create_channel
> TRACE: return connection.channel()
> TRACE: File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 144, in channel
> TRACE: return Channel(self, channel_id)
> TRACE: File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 183, in init
> TRACE: super(Channel, self).__init__(*args, **kwargs)
> TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 82, in init
> TRACE: self._x_open()
> TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 471, in _x_open
> TRACE: (20, 11), # Channel.open_ok
> TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait
> TRACE: return self.dispatch_method(method_sig, args, content)
> TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method
> TRACE: return amqp_method(self, args)
> TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 273, in _close
> TRACE: (class_id, method_id))
> TRACE: AMQPChannelException: (404, u"NOT_FOUND - no exchange '4042acd4bbec471aafc557af39ee0efe' in vhost '/'", (60, 40), 'Channel.basic_publish')
> 
> I hit this only with direct exchanges, with “auto-delete” = True.
> 
> The flow that I have is like follows:
> - node A declares a direct exchange and declares a queue
> - node A sends message to node B using a topic exchange, and embeds direct exchange name in the message
> - node B parses the message, prepares the reply, declares the (same) direct exchange and a producer
> - node B sends reply message on the producer
> - node A and node B close their channels
> - several seconds after that, when node B tries to create a new channel on the same connection (using the same channel_id), it hits this exception.
> 
> I am trying to debug this for some time, but do not have a clear way to repro this. It happens occasionally.
> 
> I understand, that since I open the channel with the same channel_id, I may receive some stale message from the server (for the same channel_id) regarding failure to publish. However, several things confuse me:
> 
> 1) The only code, which publishes to a direct exchange is as follows:
> self.exchange = kombu.entity.Exchange(name=self.exchange_name, **self.kwargs)
> self.producer = kombu.messaging.Producer(exchange=self.exchange, channel=channel, routing_key=self.routing_key)
> self.producer.publish(msg)
> How can exchange not being present here?

The Kombu producer object will declare the exchange at construction.

This is the default behavior, but it's no longer a best practice to use it.  Sadly documentation is scarce
as I have not had as much time as I'd like to update it.

The best way to use a Producer object in Kombu is to instantiate it without a default exchange,
and rather use the exchange argument to publish:

    prod = Producer(connection)
    prod.publish(message,
                            exchange=exchange,
                            routing_key=routing_key,
                            declare=[exchange],
                     )


The declare argument takes a list of entities (Queue's / Exchange's) that must declared before
publishing the message, and these are cached if possible (state belonging to the current connection, so that
they will be redeclared if the connection is lost)

Using the Producer like this also enables you to use the retry argument properly so that the declare operation
also happens before a retry:

    prod.publish(message,
                            exchange=exchange,
                            declare=[exchange],
                            retry=True,
                            retry_policy={
                                'max_retries': None,
                                'interval_start': 2.0,
                                'interval_step': 2.0,
                                'interval_max': 30.0,
                            })

The retry_policy argument can be a dict containing any of the arguments supported
by Connection.ensure_connection:

http://kombu.readthedocs.org/en/latest/reference/kombu.html#kombu.Connection.ensure_connection

> 
> 2) Even though all exchanges have auto-delete=True, I see more than 3000 existing direct exchanges on the rabbitmqctl list_exchanges output. The queues that used these exchanges (also auto-delete=True) do not exist anymore.
> When auto-delete=True exchanges are deleted? Is there some timeout on their deletion? The documentation says that there SHOULD be a reasonable timeout. What is this timeout? Can it be configured? Is exchange auto-deletion supposed to work in my version?

I'm unsure of the exact details but it has probably been mentioned before if you search the mailing-list archives.
You are not recommended to use the auto_delete flag anymore, and in fact it's no longer supported by
some clients (i.e. librabbitmq)

> 
> 3) The exchanges that I receive exceptions about do not exist indeed. How can I debug the rabbitmq server and understand why it decides to delete some exchanges, but not the other ones?