[rabbitmq-discuss] GPB and AMQP/RabbitMQ

[Elsner, Robert]

Hey Kirk,

> Ignore my previous email. This thread is updating too fast 
> for my old fingers to keep up. :-)

Luckily for me there is no ordering in SMTP, and I received that after
this one, for whatever reason. :)

> Interestingly, when GPB came out, ActiveMQ did this precise 
> thing (to have a parallel wire protocol beyond the existing 
> ones that was based on using GPB as the wire representation), 
> and they said that it worked quite well. That being said, it 
> definitely wouldn't work for AMQP given that it is designed 
> as a completely neutral wire representation, but it might be 
> useful for members of the WG to know that many people find 
> the current wire encoding a bit too high-overhead.

I did not know ActiveMQ had that, very interesting.  The part I have
enjoyed the most about GPB is that the languages we use (C#, Java,
Erlang, C) have generated source, operate very similarly, and work
incredibly well.  GPB versus our current packet format adds a little
overhead, but the flexibility and extensibility far exceeds what we've
done thus far.  I know there are other libraries out there, do you know
if anyone has done anything similar with AMQP and the other formats?

I've looked over some of the AMQPWG posts and they do seem to have the
idea that the overhead can be a little bit extreme.  Perhaps a push from
the RabbitMQ team to help work this out would be beneficial.  Hopefully
some good things will come out of their next meeting.

> The problem I've personally always had with GPB in this case 
> is that you still need to have some type of message 
> termination encoding, or some type of prefix to let you know 
> the size, as otherwise GPB will just consume data from the 
> inbound stream. So GPB in and of itself would still need some 
> other type of framing going on to say "the next thing should 
> be interpreted as per this .proto, and by the way, it's going 
> to be 1024 bytes" or something. How would you think of 
> working around that?

Luckily that problem is currently handled by AMQP.  The general solution
I've seen is to prefix everything with the size of the protobuf message.
Of course if you start listening to a stream mid-message (for whatever
reason) you first have to synchronize to the messages.

I think if I was to go to GPB as a wireformat for AMQP, I would probably
wait until multicast udp/unicast udp support were added, and use the
datagram concept of UDP as the message delimiter.  This might be
beneficial for my use, but adds a higher workload for the brokers.

The one factor I don't like about GPB is that it isn't self-describing,
to really understand a message one needs the .proto or the code
generated from it.  For something like AMQP it might be more useful to
the community to pick a binary wireformat that encodes well, which also
self-describes its message.  

Rob