[rabbitmq-discuss] Cluster Setup

Thu Jul 19 03:57:24 BST 2012

On 19 Jul 2012, at 02:15, mrajkovic wrote:
> Hi Tim,
> 
>  
> 
> Sorry it was 1am locally here when we were emailing last night, and I needed to getup for an early morning appointment….
> 
> 

Sleeping is pretty important, so fair enough! :)
>  
> 
> 
> telnet has blocked it on that port…
> 
> root at mf-01:~# telnet mf-02 4369
> 
> Trying 119.31.227.106...
> 
> telnet: Unable to connect to remote host: Connection refused
> 
> 

That's more like what we'd expect, given that epmd isn't listening other than on localhost.

>  
> 
> then I ran netstat on mf-02
> 
> root at mf-02:~# netstat -p -l --numeric-ports
> 
> Active Internet connections (only servers)
> 
> Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
> 
> tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      995/mysqld
> 
> tcp        0      0 0.0.0.0:10000           0.0.0.0:*               LISTEN      2028/perl
> 
> tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1982/apache2
> 
> tcp        0      0 127.0.0.1:4369          0.0.0.0:*               LISTEN      1868/epmd
> 

So only listening for local traffic. That's not what we should be seeing:

iske:~ root# netstat -a | grep LISTEN
tcp4       0      0  *.epmd                 *.*                    LISTEN     
tcp4       0      0  *.62693                *.*                    LISTEN     
tcp4       0      0  localhost.ipp          *.*                    LISTEN     
tcp6       0      0  localhost.ipp          *.*                    LISTEN     
iske:~ root# 

> tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN      935/dnsmasq
> 
> tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN      764/vsftpd
> 
> tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      800/sshd
> 
> tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      1755/master
> 
> tcp        0      0 0.0.0.0:39616           0.0.0.0:*               LISTEN      26602/beam.smp
> 
> tcp        0      0 119.31.227.106:5672     0.0.0.0:*               LISTEN      26602/beam.smp
> 

And clearly not all programs are having this problem. Indeed beam is running and listening happily enough.

> tcp6       0      0 :::53                   :::*                    LISTEN      935/dnsmasq
> 
> tcp6       0      0 :::22                   :::*                    LISTEN      800/sshd
> 
> tcp6       0      0 :::25                   :::*                    LISTEN      1755/master
> 
> udp        0      0 0.0.0.0:53              0.0.0.0:*                           935/dnsmasq
> 
> udp        0      0 0.0.0.0:43242           0.0.0.0:*                           1613/dccifd
> 
> udp        0      0 0.0.0.0:10000           0.0.0.0:*                           2028/perl
> 
> udp6       0      0 :::53                   :::*                                935/dnsmasq
> 
>  
> 
> it looks like its only listening on localhost and not all ports
> 
> 
Clearly. The environment variables that govern epmd's behaviour (taken from http://www.erlang.org/doc/man/epmd.html) are:

ERL_EPMD_ADDRESS
This environment variable may be set to a comma-separated list of IP addresses, in which case the epmd daemon will listen only on the specified address(es) and on the loopback address (which is implicitly added to the list if it           has not been specified). The default behaviour is to listen on all available IP addresses.

And in terms of access restrictions, the same source gives us: 

<quote>
Access restrictions
The epmd daemon accepts messages from both localhost and remote hosts. However, only the query commands are answered (and acted upon) if the query comes from a remote host. It is always an error to try to register a nodename if the client is not a process located on the same host as the epmd instance is running on, why such requests are considered hostile and the connection is immediately closed.

The queries accepted from remote nodes are:

Port queries - i.e. on which port does the node with a given name listen

Name listing - i.e. give a list of all names registered on the host

To restrict access further, firewall software has to be used.

</quote>

Finally, you can control the list of IP addresses epmd will listen on using the `-address List` flags on startup. Rabbit does not do any of these things, your environment config file isn't doing this and so I'm left a little confused. For the beam (emulator) you can set a parameter in the kernel application (inet_dist_use_interface) to control this, but you're not doing that AFAICT - you don't have any other config files (e.g., in wherever RABBITMQ_CONFIG_FILE is pointing) do you? /etc/rabbitmq can contain a rabbitmq.config file as well as the environment config file you posted originally. Is there anything in there? Even so, inet_dist_use_interface doesn't actually affect what addresses epmd will LISTEN on anyway AFAICT.

Is it possible that some other software in your environment could be starting epmd with -address *before* the rabbit nodes are started up? So really I'm at a bit of a loss here. Could you add to the mix by letting me know what Erlang/OTP version you've got installed, where it came from (i.e., apt, compiled from source, etc)? 

Having looked at the epmd source code, if there are no explicit addresses passed to the server then epmd uses INADDR_ANY, which listens on all interfaces (on most linux variants anyway). Unless you can find something in the configuration and/or environment, or there is something going on that causes INADDR_ANY to only bind to localhost (the result of muti-homing or some such) then I don't know what's going on with this one. Unless someone else has any light to shed, or you're able to locate some extraneous config/environment then we might have to take a wander over to the erlang-questions mailing list and ask there, though I will need you to come along and ask the question initially as I'm not the one who's able to replicate this behaviour.

One last question. You're not trying to run rabbit in a restricted environment (i.e., in a jail) are you? It could just be possible that security permissions are restricting the available addresses you can bind to. 

Tim

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120719/fa622b7d/attachment.htm>