[rabbitmq-discuss] Weird Crash (91MB message over STOMP) [Reproducible]

[Darien Kindlund]

> I reckon the cause is some inefficiency in the way we process stomp bodies,
> causing the server to allocate far more memory than it ought to.
>
> Shouldn't be too hard to track down and fix. Will do so hopefully this week.
>
> Thanks again for your detailed bug report and constructing such a simple
> test case. Saves us a lot of time.

No problem; glad to help.  When you're ready to test the fix on real
data, let me know.  I'd be willing to do some testing, so long as the
patch only requires recompiling the STOMP adapter (as I'm using the
pre-packaged v1.6.0 of RabbitMQ).

This particular bug makes me wonder about a much larger issue: In
general, how *should* RabbitMQ behave when it's fed an astronomically
large message size?

Let's say that RabbitMQ is smart, in that if the message is to big to
fit into RAM, it will attempt to page the rest of the message onto
disk ... but what happens when the disk is completely full and there's
simply no space left?

Ideally, I'm thinking (in this scenario) RabbitMQ should gracefully
return an error message to the producer, discard the message, and
somehow terminate the producer's send operation.  I'm not sure if this
is exactly how it's handled already -- or if this behavior is
consistent across all the different connector transports (e.g., HTTP,
STOMP).

I'm not an AMQP spec expert, but if the AMQP spec requires that all
producers supply the message size in the header (ahead of time), then
RabbitMQ could easily perform sanity checks on the maximum allowable
message size.  A RabbitMQ admin could specify this configuration
option in the /etc/rabbitmq/rabbitmq.conf file... or RabbitMQ could
approximate a maximal message size by looking at the total RAM + a
portion of the unused swap.  That way RabbitMQ could inspect the
message size and pre-emptively reject the producer's message before
running into the out-of-RAM/diskspace issue.

If the AMQP spec doesn't require producers to tell RabbitMQ the
message size in an initial header, I'm guessing that feature would be
very beneficial in a future version.

Of course, this 'message size' sanity check may become even harder to
implement, in a clustered environment.  Perhaps all cluster members
have to agree on some max threshold... or if each cluster member has
their own threshold, then the producer will have to play the odds of
getting their message sent properly -- depending on their message
size.

I'm thinking you guys have already thought/implemented something like
this... I'd be interested to know how RabbitMQ behaves currently in
this scenario as well as if that behavior is ideal.

Thanks,
-- Darien