[rabbitmq-discuss] Confirmation IDs

[Nagy, Attila]

On 07/27/14 21:30, Michael Klishin wrote:
>> Nope, from the record's identifier. I have an incoming task queue
>> (for
>> example a file which is appended with records, or a database which
>> has
>> rows), from which I can have a unique identifier which is always
>> available and can directly point to the record.
>> So counting (because publish doesn't give it back) the sent messages
>> (and resetting the counter on reconnects) and maintaining
>> counter->original message pointers are unnecessary indirections
>> to
>> achieve this functionality.
> Coming up with a good identifier for
> an arbitrary message is much harder and there's no consensus on what's the best
> way to do this.
> Especially if you try to come up with a solution for multiple protocols.
I see three approaches:
1. let the server and the client -independently- count the messages and 
use that (the current solution for AMQP in RabbitMQ)
2. let the server count (or otherwise mark them with a unique ID) the 
messages and return it to the client, so it will know about that
3. let the client specify one

I think the first has many problems, it's not flexible, the client has 
to maintain translation tables and has to count exactly the same way as 
the server.
This way the counter must be implemented in the lowest level (in the 
AMQP library, because it's the only one which knows about the connection 
state, which can reset the counter) and higher levels need to sync with 
that, otherwise they won't be able to tell which messages have been 
confirmed.
How do other libraries work in this regard?

Pika's publisher example for example is both wrong (it will die on 
reconnects due to the counter resetting) and ugly. :)
http://pika.readthedocs.org/en/latest/examples/asynchronous_publisher_example.html

I've just made a quick peek into the java client, so I may be wrong. I 
guess it has these problems too, the counter can be queried 
independently of the publishing, therefore it can't be used in multi 
threaded environments (well, at least you have to open one channel per 
thread, or put a lock on the client so there won't be a race condition 
between querying the counter and sending the message).

>> But STOMP is not really an option if you want to do serious things
>> in
>> RabbitMQ. :)
> I can't see why not.
I've tried all STOMP brokers and finally stick with RabbitMQ, because it 
was the best in many ways. But as in most of the other brokers, STOMP 
offers a limited control on what the broker does in the background, 
which is unsuitable for a lot of use cases.
For example, for me serious means exact access control. And because 
STOMP does abritrary bindings and declares, the clients can't have 
strict policies, they will have always more power than it's needed with 
AMQP.
>
>> Isn't it possible to add additional info in the ACK frame without
>> breaking current clients?
>> Or making it only appear when publish has a given pattern? (like
>> with STOMP)
> For AMQP 0-9-1 clients, it is not. STOMP header values can be of arbitrary length and many
> headers are optional. AMQP 0-9-1 methods have a fixed (mandatory) attributes
> and only *some* have optional ones ("extra arguments", as an attribute table).
> Unfortunately, basic.ack is not one of the latter group.
>
OK, then could you please consider the following?
With basic.publish setting a custom header replaces the message's 
confirm sequence number with the header's value (with type 
compatibility, so this will be a limitation, only 64(?) bit unsigned 
ints could be used) and will basic.ack with that.

I mean this would be a normal publish (if confirms are turned on, it 
will have the current counter as the sequence number):

         properties  =  pika.BasicProperties(app_id='example-publisher',
                                           headers={'key':'value'})

         self._channel.basic_publish(self.EXCHANGE,  self.ROUTING_KEY,
                                     message,
                                     properties)
         self._message_number  +=  1


And this will have 123456 as its sequence number:

         properties  =  pika.BasicProperties(app_id='example-publisher',
                                           headers={'key':'value', 'X-RabbitMQ-confirm-seq':123456})

         self._channel.basic_publish(self.EXCHANGE,  self.ROUTING_KEY,
                                     message,
                                     properties)

The consequences are to be borne by the client.