Consul: connection refused to Consul UI unless -client option is public address.
I am running consul 0.4.1 in EC2, on Amazon Linux ami-607bd917 (eu-west-1, t2.micro, naked; no yum updates), and I cannot reach the consul UI externally from the instance. My cluster bootstraps fine.
$ curl http://10.10.10.98:8500/ui/dist/
curl: (7) Failed to connect to 10.10.10.98 port 8500: Connection refused
$ curl http://127.0.0.1:8500/ui/dist/
# correct HTML response
I am using an upstart job like:
description "Consul agent"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
script
# Make sure to use all our CPUs, because Consul can block a scheduler thread
export GOMAXPROCS=`nproc`
# Get the public IP
BIND=`ifconfig eth0 | grep "inet addr" | awk '{ print substr($2,6) }'`
export CLUSTER_SIZE=$(aws autoscaling describe-auto-scaling-groups | jq -c -M -r '.AutoScalingGroups[] | {name: .AutoScalingGroupName, desired_capacity: .DesiredCapacity}' | grep je-consul-qa13-all.*ConsulCluster | jq -c -M -r '.desired_capacity' | tr -d "\n")
exec /usr/local/bin/consul agent \
-bootstrap-expect $CLUSTER_SIZE \
-config-dir="/etc/consul.d" \
-bind=$BIND \
>>/var/log/consul.log 2>&1
end script
and /etc/consul.d/consul.json as
{
"server": true,
"rejoin_after_leave": true,
"enable_syslog": true,
"syslog_facility": "local5",
"data_dir": "/var/consul/data",
"ui_dir": "/var/consul/ui",
"datacenter": "eu-west-1",
"statsd_addr": "<some IP>:8125",
"recursor": "<some IP>"
}
$ netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 10.10.10.98:tmi *:* LISTEN
tcp 0 0 10.10.10.98:amberon *:* LISTEN
tcp 0 0 10.10.10.98:8302 *:* LISTEN
tcp 0 0 localhost:cvd *:* LISTEN
tcp 0 0 localhost:fmtp *:* LISTEN
tcp 0 0 *:ssh *:* LISTEN
tcp 0 0 localhost:asterix *:* LISTEN
tcp 0 0 10.10.10.98:amberon 10.233.2.251:35382 TIME_WAIT
tcp 0 0 10.10.10.98:39957 10.233.2.251:amberon TIME_WAIT
tcp 0 0 10.10.10.98:tmi 10.233.2.251:40657 ESTABLISHED
tcp 0 0 10.10.10.98:amberon 10.233.0.36:56857 TIME_WAIT
tcp 0 512 10.10.10.98:ssh 10.10.30.248:63183 ESTABLISHED
tcp 0 0 10.10.10.98:tmi 10.233.2.251:40654 ESTABLISHED
tcp 0 0 10.10.10.98:57182 10.233.2.251:tmi ESTABLISHED
tcp 0 0 *:ssh *:* LISTEN
udp 0 0 localhost:asterix *:*
udp 0 0 10.10.10.98:51700 10.233.0.10:8125 ESTABLISHED
udp 0 0 *:bootpc *:*
udp 0 0 10.10.10.98:amberon *:*
udp 0 0 10.10.10.98:8302 *:*
udp 0 0 10.10.10.98:ntp *:*
udp 0 0 localhost:ntp *:*
udp 0 0 *:ntp *:*
udp 0 0 fe80::403:8eff:fe95:36c:ntp *:*
udp 0 0 localhost:ntp *:*
udp 0 0 *:ntp *:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 10585 /tmp/tmux-500/default
unix 7 [ ] DGRAM 8720 /dev/log
unix 2 [ ACC ] STREAM LISTENING 8799 /var/run/dbus/system_bus_socket
unix 2 [ ACC ] STREAM LISTENING 7627 @/com/ubuntu/upstart
unix 2 [ ACC ] SEQPACKET LISTENING 7832 @/org/kernel/udev/udevd
unix 3 [ ] STREAM CONNECTED 10578
unix 2 [ ] DGRAM 11277
unix 2 [ ] DGRAM 10519
unix 3 [ ] STREAM CONNECTED 10579
unix 3 [ ] DGRAM 7841
unix 3 [ ] STREAM CONNECTED 10584
unix 3 [ ] STREAM CONNECTED 8803
unix 3 [ ] STREAM CONNECTED 8802
unix 3 [ ] STREAM CONNECTED 8807
unix 3 [ ] STREAM CONNECTED 8808 /var/run/dbus/system_bus_socket
unix 2 [ ] DGRAM 10217
unix 3 [ ] STREAM CONNECTED 10523
unix 3 [ ] STREAM CONNECTED 10522
unix 3 [ ] DGRAM 7840
unix 3 [ ] STREAM CONNECTED 10581
unix 3 [ ] STREAM CONNECTED 10582
unix 2 [ ] DGRAM 9444
unix 2 [ ] DGRAM 9739
unix 3 [ ] STREAM CONNECTED 10583
I can work around this by, in my upstart, invoking consul agent .... -client=$BIND ... but then that forces me to specify the -rpc-addr=<external IP> on each consul command (in the server cluster, anyhow) thereafter; rather not do that since it complicates things a bit.
I can also work around this by running another consul cluster of agents whose only job is to join the cluster and host the UI - but I'd rather not do that, since it seems like it shouldn't be necessary and will cost more in terms of $ and management.
Is there a way to make this work without my workarounds?
petemounce
All 8 comments
You need to configure the HTTP address to listen on a public IP. By default it uses loopback. The -client flag does this, but also changes the IP of all the listeners (RPC, HTTP, DNS). You can do more fine grained configuration with a configuration file however.
armon
on 14 Jan 2015
That works perfectly; thanks!
petemounce
on 14 Jan 2015
@petemounce Could you share exactly what you did to make it work? I am also having the same problem.
saulshanabrook
on 11 May 2015
@saulshanabrook I used the configuration file to set the http interface to listen on the NIC instead of the default loopback address.
...
"addresses" : {
"http": "10.10.10.257" // obviously made up ;)
}
...
Here's the consul agent upstart task from my cloudformation:
"/etc/init/consul.conf": {
"content": {
"Fn::Join": [
"",
[
"description \"Consul agent\"\n",
"\n",
"start on runlevel [2345]\n",
"stop on runlevel [!2345]\n",
"\n",
"respawn\n",
"\n",
"script\n",
" # Make sure to use all our CPUs, because Consul can block a scheduler thread\n",
" export GOMAXPROCS=`nproc`\n",
"\n",
" # Get the public IP\n",
" BIND=`ifconfig eth0 | grep \"inet addr\" | awk '{ print substr($2,6) }'`\n",
"\n",
" export CLUSTER_SIZE=$(aws autoscaling describe-auto-scaling-groups | jq -c -M -r '.AutoScalingGroups[] | {name: .AutoScalingGroupName, desired_capacity: .DesiredCapacity}' | grep je-",
{
"Ref": "FeatureName"
},
"-",
{
"Ref": "EnvironmentName"
},
"-",
{
"Ref": "Tenant"
},
".*ConsulCluster | jq -c -M -r '.desired_capacity' | tr -d \"\\n\")\n",
" exec /usr/local/bin/consul agent \\\n",
" -bootstrap-expect $CLUSTER_SIZE \\\n",
" -config-dir=\"/etc/consul.d\" \\\n",
" -bind=$BIND \\\n",
" >>/var/log/consul.log 2>&1\n",
"end script\n",
"\n"
]
]
}
}
...
"022_configure_http_api_interface": {
"command": "BIND=`ifconfig eth0 | grep \"inet addr\" | awk '{ print substr($2,6) }'` && sed -i \"s/http\\\": \\\"\\\"/http\\\": \\\"${BIND}\\\"/\" /etc/consul.d/consul.json"
},
@petemounce thank you!
Would $BIND be 10.10.10.257 as well in this case?
With it set up like that I still get this:
[centos@ip-10-128-1-235 ~]$ curl 10.128.1.235:8500/v1/catalog/nodes
curl: (7) Failed connect to 10.128.1.235:8500; Connection refused
But works with localhost
[centos@ip-10-128-1-235 ~]$ curl localhost:8500/v1/catalog/nodes
[{"Node":"10.128.1.115","Address":"10.128.1.115"},{"Node":"10.128.1.194","Address":"10.128.1.194"},{"Node":"10.128.1.198","Address":"10.128.1.198"},{"Node":"10.128.1.218","Address":"10.128.1.218"},{"Node":"10.128.1.235","Address":"10.128.1.235"},{"Node":"10.128.1.33","Address":"10.128.1.33"},{"Node":"10.128.1.57","Address":"10.128.1.57"},{"Node":"10.128.1.73","Address":"10.128.1.73"},{"Node":"10.128.1.80","Address":"10.128.1.80"}]
This is my config:
[centos@ip-10-128-1-235 ~]$ sudo cat /etc/consul/consul.json
{
"datacenter": "tag_Service_controller",
"addresses" : {
"http": "10.128.1.235"
},
"bind_addr": "10.128.1.235",
"node_name": "10.128.1.235",
"rejoin_after_leave": true,
"domain": "consul",
"retry_join": [ "10.128.1.80", "10.128.1.235", "10.128.1.57" ],
"server": true,
"bootstrap_expect": 3,
"verify_incoming": false,
"verify_outgoing": false,
"data_dir": "/var/lib/consul",
"ui_dir": "/usr/share/consul-ui",
"disable_remote_exec": false
}
Yes, think so. My pleasure :-)
Sent from my phone. Please excuse typos and brevity, but never text speak.
On 11 May 2015 23:04, "Saul Shanabrook" [email protected] wrote:
@petemounce https://github.com/petemounce thank you! Would $BIND be
"10.10.10.257" as well in this case?—
Reply to this email directly or view it on GitHub
https://github.com/hashicorp/consul/issues/599#issuecomment-101061655.
petemounce
on 12 May 2015
[root@localhost ~]# consul agent -h | grep client
-client=127.0.0.1 Sets the address to bind for client access.
we can see : the default ip is 127.0.0.1 ,modify this val change client visit ip
eg:
consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul/ -ui-dir /root/software/consul-ui/ -client=192.168.23.154
liaokailin
on 17 Aug 2015
Hello.
I'm trying to solve similar task.
I need to connect to consul agent outside the machine using HTTP Api interface (port 8500).
The difference is - I want to connect to consul agent running in client mode, not server mode.
Is it possible?
Cause I tried different options with my consul.json config.
It's
{
"advertise_addr": "10.3.0.1",
"bind_addr": "10.3.0.1",
"addresses" : { "http": "10.3.0.1 0.0.0.0" },
"service": {"name": "myService"},
"client_addr": "0.0.0.0",
"datacenter": "westus",
"node_id": "51f55f0e-f215-44c6-95c6-3e776b9f7f0b",
"retry_join": ["10.2.10.19", "10.2.10.20", "10.2.10.21"],
"server": false,
"ui": true,
"log_level": "info",
"raft_protocol": 3,
"ports": { "dns": 53 },
"recursors": ["168.63.129.16"],
"telemetry": {
"dogstatsd_addr": "127.0.0.1:8125"
}
}
And it doesn't connect outside.
It succesfully opens http://10.3.0.1:8500/v1/kv/myKey
and http://127.0.0.1:8500/v1/kv/myKey from 10.3.0.1 machine.
But outside the machine http://10.3.0.1:8500/v1/kv/myKey can be opened.
I checked I can open http://10.2.10.19:8500/v1/kv/myKey (which consul server ip).
So is it possible to connect to consul client agent using HTTP API?
ivasilyev-servicetitan-com
on 21 Dec 2018
Related issues
wing731
·
3Comments
powerman
·
3Comments
eshujiushiwo
·
3Comments
slackpad
·
3Comments
darron
·
4Comments
Most helpful comment
That works perfectly; thanks!