[rabbitmq-discuss] DOS protection

Mon Jul 8 20:29:11 BST 2013

On Monday 8 July 2013 at 17:49, Matthias Radestock-3 [via RabbitMQ] wrote:

> Carl,  
>  
> thanks for the answers. A few follow-up questions...  
>  
> On 08/07/13 16:23, Carl Hörberg wrote:  
> > On Monday 8 July 2013 at 12:19, Matthias Radestock wrote:  
> > > - what entities do you want to monitor? You mention channels,  
> > > consumers and declarations (presumably by that you mean exchanges,  
> > > queues, and bindings). Any others?  
> >  
> >  
> > That, plus connection count.  
>  
> Connection count is bounded by the available file descriptors, for which  
> you can tune the limit in the O/S. I would have thought that is sufficient.

You're right, the global limit, but we monitor and limit individual vhost connections too. The only problem is if a user creates a lot of connections within 5 seconds, then we can't block new connections fast enough.  

And the way we block is suboptimal too, today we remove the password (or remove the vhost permission), but then they can't access the mgmt interface either, which confuses a lot of users.  
>  
>  
> > A global limit would be sufficient for the time being, just for  
> > protection, so that the cluster doesn't dies before we can see who  
> > has done what.  
>  
>  
> So say rabbit enforced global limits for the numbers of channels,  
> consumers, exchanges, queues and bindings. How would you go about  
> picking the limits?  
>  
> There is nothing wrong with having lots of channels, or lots of  
> consumers, or lots of exchanges, etc, etc. But if you set upper bounds  
> on what rabbit can sustain for each one of them, then a rabbit which  
> encounters mixed usage could still run out of memory.

True.. Most often it's not that complicated though. Often we can see that user uses a lot of resources, and then we contact that user and make sure that he's doing the right thing. Again, the problem we have is when customers by bugs or by other means creates a lot of something very very fast, faster than we can monitor.  
>  
>  
> Furthermore, as you know, in the default configuration messages take up  
> some residual memory even when paged to disk, which means there is less  
> memory available for everything. This too makes it hard to choose any  
> bounds which have a reasonable chance of working w/o being silly low.  
>  
True, i've seen the message index grow to many gigabytes, which is way we now monitor queue length too, but "fortunately" message queues can only grow with a couple of thousands messages per second, not millions, which is required to build a huge index.  
>  
>  
> Since your main concern is memory exhaustion - and by that presumably  
> you mean rabbit crashing because it cannot allocate any more memory,  
> rather than just hitting the memory alarm and blocking publisher -  
> perhaps a better - simpler and more versatile - behaviour would be for  
> rabbit to block all channel, consumer and resource declarations when  
> memory gets *really* tight.

That would be a great solution! Or somehow "slow things down", so we have the chance to use the mgmt api and read the values before it's too late.    
>  
>  
>  
> Regards,  
>  
> Matthias.  
> _______________________________________________  
> rabbitmq-discuss mailing list  
> [hidden email] (/user/SendEmail.jtp?type=node&node=27904&i=0)  
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>  
>  
> If you reply to this email, your message will be added to the discussion below: http://rabbitmq.1065348.n5.nabble.com/DOS-protection-tp27876p27904.html  
> To unsubscribe from DOS protection, click here (http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=27876&code=Y2FybC5ob2VyYmVyZ0BnbWFpbC5jb218Mjc4NzZ8LTEyNDcxMDc4NjM=).
> NAML (http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml)  

--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/DOS-protection-tp27876p27906.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130708/dd017519/attachment.htm>