[rabbitmq-discuss] Http Management Plugin: statistics_db_node not_running

[Simon MacMullen]

On 22/01/2014 17:02, Ben J. Heiskell wrote:
> I'm guessing that through sheer bad luck, the node which crashed was
> the active statistics_db_node. What I don't understand is why I had
> to manually intervene to get the statistics db back online. Is this a
> bug? What is the expected behavior here?

The stats DB *should* fail over to any other node which has the full 
management plugin (not just the agent) enabled. And indeed that's what I 
see when starting and stopping nodes in a cluster :-/ As you note we 
have seen the occasional report of this behaviour, but I can't replicate it.

> - When visiting the management plugin from a browser, the overview
> tab displays the error message: "TypeError: Cannot read property
> 'connections' of undefined". Additionally, the connections and
> channels tabs are both empty. Exchanges and Queues are populated as
> expected, but the statistics related columns (rates and counts) are
> all empty. Clicking on an exchange results errors with
> "ReferenceError: exchange is not defined". Clicking on a queue errors
> with "TypeError: Cannot read property 'ram_msg_count' of undefined".

Yes, the web UI will be quite broken without a stats DB.

> - Everything aside from the management API appears to have continued
> to work correctly.

Yes, it doesn't affect anything else.

>  - I found the following message in the crashed host's logs, which
> repeated once for every host in the cluster:
>
> =ERROR REPORT==== 18-Jan-2014::17:59:26 ===
> Mnesia(rabbit at rabbitmq04): ** ERROR ** mnesia_event got
> {inconsistent_database, starting_partitioned_network,
> rabbit at rabbitmq05}

That's evidence of a partitioned network. It's quite possible (even 
likely) that the stats DB failure is related to this. I will 
investigate, but don't hold your breath; this is difficult to replicate.

Cheers, Simon