[rabbitmq-discuss] [Q] best way to add a sequencer to the broker

[Chuck Remes]

On Dec 29, 2008, at 10:18 AM, Ben Hood wrote:

>> 2. Is there a better way to accomplish this task?
>
> I'm not quite sure I fully understand the actual use case. AFAICT you
> want each message to have it's own sequence number that you want to
> set yourself but have some kind of intermediary increment this number.
> But because this is stateless operation (because you are just
> incrementing a context free number), I must conclude that I don't
> really understand what you are really trying to achieve (although I've
> read through your post twice - maybe I'm just being stupid). Or is
> that the publisher always sets this sequence number to 0 and you want
> to stamp it with a globally incremented number?

Sorry it was still unclear; I rewrote the email twice before hitting  
the send button. :)

Your last sentence quoted above gets it exactly. Each publisher will  
set the sequence number to 0 so that the republisher/sequencer service  
can add the global number stamp to it. I guess the publishers could  
also set it to nil; the initial value is irrelevant since the  
Sequencer will overwrite it.

The use case is to stamp a global order on all messages through the  
broker for the purposes of replay, debugging and async logging. I  
realize that during normal operation the services will be publishing  
messages in an arbitrary order; it doesn't matter to me. I need to  
recreate that arbitrary order *exactly* for a replay.

Think of the sequencer as the software equivalent of a hardware bus  
clock. Alternately, think of the sequencer as the equivalent of adding  
an identity column to a database table (this column contains a  
monotonically increasing integer assigned by the server).

> BTW I don't whether the delivery tag that is sent in the header of the
> message to the consumer is of any use to you for sequencing.

I don't plan to use the delivery tag at all. When I republish, I need  
to pass through the routing key from the original message's header.

>> I've read some notes
>> about future releases of rabbitmq allowing for "internal" clients  
>> that
>> run inside the rabbitmq memory space. I think this is an ideal use of
>> that capability. Am I right or barking up the wrong tree? And does
>> that internal client need to be written in erlang or can it be java/ 
>> c/
>> whatever?
>
> The Erlang AMQP library can run transparently in the address space of
> the broker. The interface is exactly the same as the client library
> that uses TCP, the only difference is that the TCP stack and the
> message serialization/deserialization is avoided. This basically takes
> advantage of the fact that Erlang messages natively and hence you can
> unobtrusively abstract out the AMQP wire format.
>
> The types of things that this is useful for is processing messages in
> a custom fashion without a network hop and you can stick to the AMQP
> model rather than having to write something that is Rabbit specific.
> This also takes advantage of the fact that Erlang processes are
> isolated from each other and so it is ok to run "foreign code" in the
> same memory space as the broker.
>
> But I don't know if this solves your problem, because as I stated
> above, I didn't really understand it.

I think this does solve the issue. I don't know Erlang yet so I'll  
probably keep the Sequencer as an external Java client for the short- 
term, but it's nice to know I can get a performance win with a rewrite  
in Erlang. Avoiding the TCP overhead, even if it is short circuited on  
a localhost, is a good thing.

>> 3. Is there a way to make this more dynamic so I do not have to
>> declare all the exchanges up-front in this sequencer code? Ideally
>> this service could detect that new exchanges were declared by other
>> services and automatically subscribe to them to do the sequence
>> stamping and routing.
>
> I don't know if the terminology is right here, because you generally
> don't subscribe to an exchange, rather to a queue.

You are right. I was confusing the terminology. Let me try again.

As other services declare exchanges and publish to them, I would like  
my Sequencer to dynamically detect the existence of a new Exchange so  
it can create a queue, bind the new queue to the Exchange and receive  
messages from it for processing and republishing.

cr