[rabbitmq-discuss] Enabling hipe_compile in RHEL 6

[Simon MacMullen]

On 30/04/12 10:38, Emile Joubert wrote:
>>   I've changed the frame_max setting but that doesn't appear to
>> help. I've tried increasing the number of workers across additional
>> hardware as well, but Rabbit still seems to cap at 20k message/sec. When
>
> The maximum framesize allows you to choose between better latency (small
> framesize) or throughput (large framesize). If many messages are larger
> than 128Kb then increasing framesize may improve throughput (provided
> consumers can keep up).

(But for small messages it makes no difference.)

> When you increased the number of workers, were you able to determine
> whether the load was effectively spread amongst all of them? The QoS
> prefetch count setting will help to ensure fair distribution. Try a
> small number (e.g. 10) as a starting point for tuning this value.
>
>> I set hipe_compile to true, the systems states 'Not HiPE compiling: HiPE
>> not found in this Erlang installation.' What's odd is
>> that erlang-hipe-R14B-04.1.el6.x86_64 is installed. Is there something
>> else that needs to be done for RHEL systems to enable hipe_compile? I
>
> That message means that your installation of Erlang lacks the hipe.beam
> file in the code loading path. The name of the package that includes
> this file depends on how Erlang was packaged in your system. Making use
> of HiPE won't address the core problem though.

Well, HiPE will help if a CPU-bound RabbitMQ is in fact the bottleneck. 
It won't help if RabbitMQ is IO-bound, or not the bottleneck.

But HiPE is two parts; there are support libraries in the HiPE package, 
but there also needs to be support in the VM - and I've just had a quick 
look at Peter's Erlang packages installed on my CentOS 5 box and it 
looks like HiPE support is not compiled into the VM. (You should see 
"[hipe]" in the banner displayed when starting erl.) That's not the same 
RPM as you're using, but maybe it comes from the same source?

>> think that it may help get beyond the current problem. If that doesn't
>> help, are there other settings or something I can look at to determine
>> where the bottleneck is? The RMQ server is 60% idle, doesn't have a
>> large amount of I/O wait, and doesn't seem to be saturating its network
>> cards (the server has a bonded ethernet interface). The worker machines
>> are relatively idle as well.
>
> Are *all* the workers idle, or are a small number taking all the load?
> Uneven worker load is a potential cause for the problem you describe and
> can be addressed using prefetch count:
>
> http://www.rabbitmq.com/amqp-0-9-1-reference.html#basic.qos.prefetch-count

This is certainly suspicious. Something somewhere in your system ought 
to be running at 100% capacity. If your queues are growing it should be 
the workers, but the other possibility is that RabbitMQ is not able to 
push out messages fast enough to the consumers. If so you'll be happy to 
hear that the imminent 2.8.2 release has some performance improvements 
in that area.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware