[rabbitmq-discuss] [Q] best way to add a sequencer to the broker
cremes.devlist at mac.com
Mon Dec 29 17:43:16 GMT 2008
On Dec 29, 2008, at 10:18 AM, Ben Hood wrote:
>> 2. Is there a better way to accomplish this task?
> I'm not quite sure I fully understand the actual use case. AFAICT you
> want each message to have it's own sequence number that you want to
> set yourself but have some kind of intermediary increment this number.
> But because this is stateless operation (because you are just
> incrementing a context free number), I must conclude that I don't
> really understand what you are really trying to achieve (although I've
> read through your post twice - maybe I'm just being stupid). Or is
> that the publisher always sets this sequence number to 0 and you want
> to stamp it with a globally incremented number?
Sorry it was still unclear; I rewrote the email twice before hitting
the send button. :)
Your last sentence quoted above gets it exactly. Each publisher will
set the sequence number to 0 so that the republisher/sequencer service
can add the global number stamp to it. I guess the publishers could
also set it to nil; the initial value is irrelevant since the
Sequencer will overwrite it.
The use case is to stamp a global order on all messages through the
broker for the purposes of replay, debugging and async logging. I
realize that during normal operation the services will be publishing
messages in an arbitrary order; it doesn't matter to me. I need to
recreate that arbitrary order *exactly* for a replay.
Think of the sequencer as the software equivalent of a hardware bus
clock. Alternately, think of the sequencer as the equivalent of adding
an identity column to a database table (this column contains a
monotonically increasing integer assigned by the server).
> BTW I don't whether the delivery tag that is sent in the header of the
> message to the consumer is of any use to you for sequencing.
I don't plan to use the delivery tag at all. When I republish, I need
to pass through the routing key from the original message's header.
>> I've read some notes
>> about future releases of rabbitmq allowing for "internal" clients
>> run inside the rabbitmq memory space. I think this is an ideal use of
>> that capability. Am I right or barking up the wrong tree? And does
>> that internal client need to be written in erlang or can it be java/
> The Erlang AMQP library can run transparently in the address space of
> the broker. The interface is exactly the same as the client library
> that uses TCP, the only difference is that the TCP stack and the
> message serialization/deserialization is avoided. This basically takes
> advantage of the fact that Erlang messages natively and hence you can
> unobtrusively abstract out the AMQP wire format.
> The types of things that this is useful for is processing messages in
> a custom fashion without a network hop and you can stick to the AMQP
> model rather than having to write something that is Rabbit specific.
> This also takes advantage of the fact that Erlang processes are
> isolated from each other and so it is ok to run "foreign code" in the
> same memory space as the broker.
> But I don't know if this solves your problem, because as I stated
> above, I didn't really understand it.
I think this does solve the issue. I don't know Erlang yet so I'll
probably keep the Sequencer as an external Java client for the short-
term, but it's nice to know I can get a performance win with a rewrite
in Erlang. Avoiding the TCP overhead, even if it is short circuited on
a localhost, is a good thing.
>> 3. Is there a way to make this more dynamic so I do not have to
>> declare all the exchanges up-front in this sequencer code? Ideally
>> this service could detect that new exchanges were declared by other
>> services and automatically subscribe to them to do the sequence
>> stamping and routing.
> I don't know if the terminology is right here, because you generally
> don't subscribe to an exchange, rather to a queue.
You are right. I was confusing the terminology. Let me try again.
As other services declare exchanges and publish to them, I would like
my Sequencer to dynamically detect the existence of a new Exchange so
it can create a queue, bind the new queue to the Exchange and receive
messages from it for processing and republishing.
More information about the rabbitmq-discuss