[rabbitmq-discuss] Missing features making me look at moving off RabbitMQ

[Christian Legnitto]

Thanks for the reply! Comments inline.

On Oct 29, 2010, at 6:46 PM, Matthias Radestock wrote:

> Christian,
> 
> Christian Legnitto wrote:
>> Just thought I'd send a note detailing why I am investigating moving off RabbitMQ.
> 
> Thanks for letting us know. Let's see whether we can help you...
> 
>> -- Built-in way to deal with the "slow consumer" problem
>> * For pulse, it is entirely conceivable that consumers will check in once and perhaps go away for weeks (or maybe forever) * Ideally
>> support would be configurable and similar to http://activemq.apache.org/slow-consumer-handling.html
> 
> RabbitMQ has several ways of dealing with consumers that go away:
> - exclusive queues - go away when the connection dies

I want consumers to be able to get what they missed while away. For example, if someone is running a tool on a laptop or their personal server goes down, I want the messages missed to persist.

> - auto-delete queues - go away when the last consumer disappears

Again, as above I want missed messages from the point the consumer checked in to the point they come back (which could be weeks)

> - queues with leases - go away when unused for some time

Same deal.

> - per-queue message ttl (in the next release) - messages expire after a
> per-queue configured ttl

Same deal.

> 
> Would any of these work for your use case?

Nope :-( Basically I want to give a reasonable expectation to consumers, something like "If you do not actively consume messages the most that will be kept are the latest 20,000. If the system isn't resource constrained we may store more, but you can't rely on it." I also want to be able to lift/modify this restriction for certain critical consumers. 

> 
>> -- Built-in HA
>> * It'd be nice to not have to cobble together multiple pieces to get this
> 
> Instead of "cobbled together" I'd like to think of it as "using a combination of tried & trusted best-of-breed products" ;)
> 
> We are working on better and more integrated HA. Meanwhile, the instructions for the existing active/passive HA may look a bit
> daunting, but it's actually not that hard to set up.

Yeah, honestly this was just put here for completeness. I haven't looked into it too much, but from perusing the docs + other broker docs it looks more involved than other setups.

> 
>> -- Support custom authentication schemes
>> * I'd like to be able to hook authentication to different backends (like ldap, use bugzilla for auth, etc) * I need this as I don't want
>> to maintain lists and want to give permissions based on ldap groups
> 
> This has been requested a few times but hasn't yet bubbled up far enough
> our priority list to make it into the code base.
> 
> Tweaking the code s.t. it supports your particular authentication scheme
> should only take a few hours. Implementing a more generally
> applicable and pluggable mechanism would take a few days.

I figured it wouldn't be too difficult to do, just wasn't sure about the supporting erlang libs. In my use-case I'd want to define fallback schemes as well. That is, first try to find the person in ldap, then check a local, hard-coded list of users.

>> -- Support message-level permissions
>> * It'd be nice for a publisher to be able to include a security token
>> and have the broker decide to vend to a consumer or not based on it
>> * Haven't really thought about this in-depth yet, but the problem we are trying to solve is publishing multiple-tiers of security-sensitive information where the message content itself determines who should see it
> 
> That sounds like a pretty hard problem. Are there any messaging systems
> out there that do something close to what you need here?

I don't think so. The main problem I am trying to solve is a secure way to publish bugzilla messages into the broker. As you can imagine, each message has differing levels of security, and each consumer / queue needs to only be able to see certain stuff. I need to think about the problem some more, but if I could include some key/vaue pairs from bugzilla in the message and the broker could choose to vend or not based on some configuration I believe it would be all that is needed to solve. Of course this would kill throughput...

> 
>> -- Support for WebSockets
>> * It'd be nice for there a plugin to support websockets * More important for Mozilla as most tools are web-based *
> 
> I'd be surprised if some websockets support didn't find its way into
> a rabbit plug-in soon.
> 
> Meanwhile, would any of existing Rabbit plug-ins/extensions for
> messaging-over-HTTP, such as the jsonrpc-channel
> (http://www.rabbitmq.com/plugins.html#rabbitmq-jsonrpc-channel) and
> rabbithub (http://github.com/tonyg/rabbithub) work for you?

Yep. I've also set up orbit in front of the STOMP plugin, so I can get the WebSockets functionality with additional effort.

> 
>> -- Public bugtracker
>> * Be nice to file bugs, learn from other's bugs, etc
> 
> There is no official public rabbit bug tracker but you can (and people
> occasionally do) file bugs in the rabbit bug trackers on github, ubuntu, debian, etc. That doesn't give you much insight into what the rabbit team is working on though, which is what I think you are after.


Yep, it'd be nice to cc myself on bugs, see which bugs have movement, etc.

> 
>> * I understand there is a private one. Can you just set all existing
>> bugs to restricted and provide public access? All new bugs could be
>> public and as soon as old bugs are vetted you could open them up
> 
> Something like that will probably happen eventually. The issue tracking system plays a pivotal role in the work of the RabbitMQ team. We have more than 2500 bugs recorded in it, touch about 30 bugs a day and open ~100 new ones per month. So there's a lot going on, which of course would make it an incredibly valuable resource for the community. But because it is so important to the rabbit team any fundamental changes to it require careful consideration.

Totally understandable. Bugzilla is the basis for all workflows here at Moz so we understand how important it is to make sure it's solid. That being said, we do have private bugs (secure/ corporation-specific/legal) as well as public bugs, so the end-goal is certainly doable.

Plus, once you get a public bugzilla you can use my extension to send messages to Rabbit. Once that data is flowing there are tons of cool tools you can build on top to help your org ;-)


> 
>> -- Public roadmap
>> * Very critical when you are basing your entire organization on a product
> 
> Our to do list so far has resisted attempts to condense it into a roadmap that can be navigated without a personal guide ;) And our plans are quite fluid since the prioritisation of work items is often finely balanced and can change radically with just a single user request (e.g. for pluggable authentication ;), bug report, etc.
> 
> Having said that, we are always happy to discuss our current plans with potential customers / users. Shall I give you a ring?

Eventually, yes :-) I'm knee-ddep in Firefox releases at the moment so my Rabbit stuff has been put on the back-burner.

> 
>> * Management console / API was on this list, then all of a sudden someone started working on it (thanks!), but it would have been nice to know it was coming
> 
> That is one example of the fluidity of our plans. Improvements to management had been on our todo list for a long time. It took a combination of various factors and events to push it near the top of our priority list. Once it had ended up there, exploratory work got underway very quickly. The first line of code was written on July 12th. We announced it to the world less than four weeks later, on August 6th, once we were confident it would make it into the product soon.
> 
> So if it looked like rabbitmq-management appeared out of the blue then that is because it did :)
> 
> Most features / fixes we decide to work on make it into the product within 1-2 months. Or they get parked or abandoned. For the few items that take several months we could perhaps be a bit more vocal about their progress. But doing so always carries the risk of disappointing people when the features subsequently get delayed or abandoned.
> 
> 
> Regards,
> 
> Matthias.