Ingress-nginx: bind() to 0.0.0.0:80 failed(13: Permission denied)

Created on 6 Mar 2019  ·  9Comments  ·  Source: kubernetes/ingress-nginx

NGINX Ingress controller version:
0.23.0
Kubernetes version:
1.13.3
OS:Centos7

What is the cause of this problem？
“bind() to 0.0.0.0:80 failed(13: Permission denied)”

As follows：

NGINX Ingress controller
Release: 0.23.0
Build: git-be1329b22

Repository: https://github.com/kubernetes/ingress-nginx

W0306 10:24:53.440586 6 flags.go:213] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: nginx/1.15.9
W0306 10:24:53.443177 6 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0306 10:24:53.443525 6 main.go:200] Creating API client for https://172.30.0.1:443
I0306 10:24:53.451994 6 main.go:244] Running in Kubernetes cluster version v1.13 (v1.13.3) - git (clean) commit 721bfa751924da8d1680787490c54b9179b1fed0 - platform linux/amd64
I0306 10:24:53.607310 6 nginx.go:261] Starting NGINX Ingress controller
I0306 10:24:53.610611 6 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"28f7f279-3ff9-11e9-9709-fa7a9a0b8d00", APIVersion:"v1", ResourceVersion:"145115", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
I0306 10:24:53.610634 6 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"28f75a24-3ff9-11e9-9709-fa7a9a0b8d00", APIVersion:"v1", ResourceVersion:"145114", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
I0306 10:24:53.611129 6 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"28f6b9a1-3ff9-11e9-9709-fa7a9a0b8d00", APIVersion:"v1", ResourceVersion:"145113", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/nginx-configuration
I0306 10:24:54.807925 6 nginx.go:282] Starting NGINX process
I0306 10:24:54.808024 6 leaderelection.go:205] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I0306 10:24:54.808364 6 controller.go:172] Configuration changes detected, backend reload required.
I0306 10:24:54.812553 6 leaderelection.go:214] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0306 10:24:54.812617 6 status.go:148] new leader elected: nginx-ingress-controller-797b884cbc-jx4gm
W0306 10:24:54.813959 6 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}, err services "ingress-nginx" not found
W0306 10:24:54.820366 6 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}, err services "ingress-nginx" not found
W0306 10:24:54.826694 6 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}, err services "ingress-nginx" not found
W0306 10:24:54.833291 6 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}, err services "ingress-nginx" not found
W0306 10:24:54.839729 6 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}, err services "ingress-nginx" not found
W0306 10:24:54.846318 6 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}, err services "ingress-nginx" not found
W0306 10:24:54.852754 6 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}, err services "ingress-nginx" not found
E0306 10:24:54.855134 6 controller.go:184] Unexpected failure reloading the backend:

Error: exit status 1
2019/03/06 10:24:54 [notice] 40#40: ModSecurity-nginx v1.0.0
nginx: the configuration file /tmp/nginx-cfg786210603 syntax is ok
2019/03/06 10:24:54 [emerg] 40#40: bind() to 0.0.0.0:80 failed (13: Permission denied)
nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)
nginx: configuration file /tmp/nginx-cfg786210603 test failed

W0306 10:24:54.855158 6 queue.go:130] requeuing initial-sync, err

Error: exit status 1
2019/03/06 10:24:54 [notice] 40#40: ModSecurity-nginx v1.0.0
nginx: the configuration file /tmp/nginx-cfg786210603 syntax is ok
2019/03/06 10:24:54 [emerg] 40#40: bind() to 0.0.0.0:80 failed (13: Permission denied)
nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)
nginx: configuration file /tmp/nginx-cfg786210603 test failed

picture weifan01

Most helpful comment

@sahirug my guess is that you relocated your root and haven't copied xattrs and therefore capabilities fail -> since you likely no longer have the source files, I simply recommend deleting both the container and the image and pulling it anew.

picture tgerczei on 24 Oct 2019
❤6

All 9 comments

@weifan01 please post the output of docker info from the node where the pod is running

aledbf picture aledbf on 6 Mar 2019

@weifan01 please post the output of docker info from the node where the pod is running

Containers: 26
Running: 8
Paused: 0
Stopped: 18
Images: 9
Server Version: 18.09.2
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 09c8266bf2fcf9519a651b04ae54c967b9ab86ec
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.2.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.51GiB
Name: 172-21-11-52
ID: EY67:SELK:52O7:FERH:6UTR:KLHF:JAZA:MEAJ:U4UF:ATR4:VPVJ:LHAR
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://registry.docker-cn.com/
https://docker.mirrors.ustc.edu.cn/
Live Restore Enabled: false
Product License: Community Engine

weifan01 picture weifan01 on 6 Mar 2019

Storage Driver: vfs

You are using a storage driver that does not provide support for xattr. This is required to use CAP_NET_BIND_SERVICE and be able to run as a user binding to privileged ports.

Please change the storage driver to overlay or overlay2 (the defaults) or change the ingress controller deployment to run as root.

aledbf picture aledbf on 6 Mar 2019
❤4

I'm having a similar issue on my bare metal kubernetes cluster. The error stack trace is:

Error: exit status 1 nginx: the configuration file /tmp/nginx-cfg734622380 syntax is ok 2019/08/14 06:02:44 [emerg] 96#96: bind() to 0.0.0.0:80 failed (13: Permission denied) nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied) nginx: configuration file /tmp/nginx-cfg734622380 test failed

The docker info output of the node is:

Client:
Debug Mode: false

Server:
Containers: 21
Running: 14
Paused: 0
Stopped: 7
Images: 44
Server Version: 19.03.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.18.0-1024-azure
Operating System: Ubuntu 18.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.64GiB
Name: AZOPS02
ID: KTHU:XEJG:G6CV:ILOP:BYMN:N6KD:LWEI:ZAMB:442S:FVAG:AOX7:C5KR
Docker Root Dir: /mountdisk/user/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
10.62.194.7:5000
127.0.0.0/8
Live Restore Enabled: false

Any ideas? @aledbf

sahirug picture sahirug on 14 Aug 2019

@sahirug my guess is that you relocated your root and haven't copied xattrs and therefore capabilities fail -> since you likely no longer have the source files, I simply recommend deleting both the container and the image and pulling it anew.

tgerczei picture tgerczei on 24 Oct 2019
❤6

Same issue on my cluster , I don't need to bind ingress ports 80 and 443 on the host , but i just want to run ingress and then create a nodePort service to point to container ports 80 and 443 , do I still need capabalities and privileges for that?

iahmad-khan picture iahmad-khan on 2 Dec 2019

Hi,

It can be caused by docker don't have rights to open pod's http/https ports as non-root user.
Modifing unprivilleged ports on node ( VM ) is USELESS, because problem is probably on side of POD network.

try to add :
hostNetwork: true

into
deployments/daemon-set/nginx-ingress.yaml,

or
deployments/deployment/nginx-ingress.yaml,

    17     spec:
    18       serviceAccountName: nginx-ingress
    19       hostNetwork: true
    20       containers:
    21       - image: nginx/nginx-ingress:1.6.0
    22         name: nginx-ingress
    23         ports:
    24         - name: http
    25           containerPort: 80
    26           hostPort: 80
    27         - name: https
    28           containerPort: 443
    29           hostPort: 443

Then try to run deployment / daemon set.
This resolved my problem, the nginx-ingress started and the error disappeared from the logs

dgmrdr picture dgmrdr on 20 Jan 2020

See aegershman solution in https://github.com/helm/charts/issues/15994. It worked for me.

cbdchang picture cbdchang on 28 Jan 2020

image: nginx/nginx-ingress:1.6.0

@dgmrdr that image is from a different project.

aledbf picture aledbf on 28 Jan 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

lachlancooper picture lachlancooper  ·  3Comments
kfox1111 picture kfox1111  ·  3Comments
silasbw picture silasbw  ·  3Comments
smeruelo picture smeruelo  ·  3Comments
sophaskins picture sophaskins  ·  3Comments