Aws-load-balancer-controller: K8S Network policy applied is not allowing traffic

Created on 19 Jul 2019  路  7Comments  路  Source: kubernetes-sigs/aws-load-balancer-controller

We are using ALB ingress controller for creating load balancers. Now, our requirement is that, pods should be able to receive calls from only within the namespace and from loadbalancer. But, we are not able to configure ALB with a network policy.

ALB is deployed in 'kube-system', and our pod is deployed in a namespace 'one'.

Our Network policy looks like this: 

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  namespace: one
  name: deny-from-other-namespaces
spec:
  podSelector:
    matchLabels:
  ingress:
  - from:
    - podSelector: {}
    - namespaceSelector:
        matchLabels:
          test-label: kube-system

The pods are now unreachable from the load balancer.

Is this configuration even possible from ALB?

lifecyclrotten
Source
picture scrumteamwhitewalkers
👍1

Most helpful comment

whoever stumbles upon this, we achieved this on EKS with Calico with the following:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-from-other-namespaces
  namespace: your-namespace
spec:
  podSelector:
    matchLabels:
  ingress:
  - from:
    - podSelector: {}
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-external
  namespace: your-namespace
spec:
  podSelector:
    matchLabels:
      app: your-app-label
  ingress:
  - from: []
    ports:
    - port: 3000

The first policy denies traffic from other namespaces. However, if you're using alb-ingress-controller with alb.ingress.kubernetes.io/target-type: instance (default), this will break your services, because of NodePort. To solve that, allow ingress to your app port in the second policy.

While you've restricted pod to pod traffic, the pods in other namespaces can still hit the service via node-ip:port. This is good enough for our setup, since these services are exposed to the public via ALB anyway.

If you want to restrict even that, look at denying all egress in all namespaces and only whitelisting the ones the require. The other solution is to change alb.ingress.kubernetes.io/target-type to ip, however, you'll introduce downtime to your deployments, see #814 for more details.

picture aalimovs on 23 Jan 2020
👍4

All 7 comments

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot picture fejta-bot on 17 Oct 2019

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot picture fejta-bot on 16 Nov 2019

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot picture fejta-bot on 16 Dec 2019

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot picture k8s-ci-robot on 16 Dec 2019

whoever stumbles upon this, we achieved this on EKS with Calico with the following:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-from-other-namespaces
  namespace: your-namespace
spec:
  podSelector:
    matchLabels:
  ingress:
  - from:
    - podSelector: {}
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-external
  namespace: your-namespace
spec:
  podSelector:
    matchLabels:
      app: your-app-label
  ingress:
  - from: []
    ports:
    - port: 3000

The first policy denies traffic from other namespaces. However, if you're using alb-ingress-controller with alb.ingress.kubernetes.io/target-type: instance (default), this will break your services, because of NodePort. To solve that, allow ingress to your app port in the second policy.

While you've restricted pod to pod traffic, the pods in other namespaces can still hit the service via node-ip:port. This is good enough for our setup, since these services are exposed to the public via ALB anyway.

If you want to restrict even that, look at denying all egress in all namespaces and only whitelisting the ones the require. The other solution is to change alb.ingress.kubernetes.io/target-type to ip, however, you'll introduce downtime to your deployments, see #814 for more details.

aalimovs picture aalimovs on 23 Jan 2020
👍4

@aalimovs : Do you have any idea on how to achieve this while target-type is set to IP ?. We don't have a way to define a way to allow incoming ingress traffic from ALB.

hasinthaindrajee picture hasinthaindrajee on 25 May 2020

whoever stumbles upon this, we achieved this on EKS with Calico with the following:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-from-other-namespaces
  namespace: your-namespace
spec:
  podSelector:
    matchLabels:
  ingress:
  - from:
    - podSelector: {}
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-external
  namespace: your-namespace
spec:
  podSelector:
    matchLabels:
      app: your-app-label
  ingress:
  - from: []
    ports:
    - port: 3000

The first policy denies traffic from other namespaces. However, if you're using alb-ingress-controller with alb.ingress.kubernetes.io/target-type: instance (default), this will break your services, because of NodePort. To solve that, allow ingress to your app port in the second policy.

While you've restricted pod to pod traffic, the pods in other namespaces can still hit the service via node-ip:port. This is good enough for our setup, since these services are exposed to the public via ALB anyway.

If you want to restrict even that, look at denying all egress in all namespaces and only whitelisting the ones the require. The other solution is to change alb.ingress.kubernetes.io/target-type to ip, however, you'll introduce downtime to your deployments, see #814 for more details.

Hello I have tried this with NodePort but I can still connect from another other pod from another namespace to this restricted pod's port.

kubectl get pods -n os  -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE                                           NOMINATED NODE   READINESS GATES
nginx-5976c8f8d6-s674r   1/1     Running   0          15h   10.100.149.18   ip-10-100-150-255.eu-west-1.compute.internal   <none>           <none>
nginx-5976c8f8d6-xw5vb   1/1     Running   0          16h   10.100.97.105   ip-10-100-102-177.eu-west-1.compute.internal   <none>           <none>

and then on on default namespace (not os namespace) 

kubectl run --generator=run-pod/v1  tmp-shell  --rm -i --tty --image nicolaka/netshoot -- /bin/bash
If you don't see a command prompt, try pressing enter.
bash-5.0# nc -zv 10.100.149.18 80
Connection to 10.100.149.18 80 port [tcp/http] succeeded!
bash-5.0# nc -zv 10.100.97.105 80
Connection to 10.100.97.105 80 port [tcp/http] succeeded!

if i remove allow-external it refuses connection (calico works!) from default namespace but also ALB does not work because healtchecks fails on ALB (ingress ALB) so the scenario fails. What i want is to only allow traffic from ALB Ingress and from Same namespace. Also my ALB Ingress resides on same Namespace ( os namespace) with the pods. What i have noticed is that even healt check fails for all targets (namely node:Port ) traffic goes thru and reaches - as this is default behaviour, if no target is reachable send traffic anyway-. The traffic comes from ALB Igress' Internal IP AND Healtchecks are done on SAME ALB Ingress Internal IPs too. 

nginx-5976c8f8d6-s674r nginx 10.100.150.255 - - [06/Jun/2020:08:32:13 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "149.202.81.xx"
nginx-5976c8f8d6-xw5vb nginx 10.100.150.255 - - [06/Jun/2020:08:32:14 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "149.202.81.xx"
nginx-5976c8f8d6-xw5vb nginx 10.100.150.255 - - [06/Jun/2020:08:32:15 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "149.202.81.xx"
nginx-5976c8f8d6-xw5vb nginx 10.100.102.177 - - [06/Jun/2020:08:32:15 +0000] "GET / HTTP/1.1" 200 612 "-" "ELB-HealthChecker/2.0" "-"
nginx-5976c8f8d6-s674r nginx 10.100.129.196 - - [06/Jun/2020:08:32:15 +0000] "GET / HTTP/1.1" 200 612 "-" "ELB-HealthChecker/2.0" "-"
nginx-5976c8f8d6-xw5vb nginx 10.100.150.255 - - [06/Jun/2020:08:32:15 +0000] "GET / HTTP/1.1" 200 612 "-" "ELB-HealthChecker/2.0" "-"
nginx-5976c8f8d6-s674r nginx 10.100.150.255 - - [06/Jun/2020:08:32:16 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "149.202.81.xx"
nginx-5976c8f8d6-s674r nginx 10.100.150.255 - - [06/Jun/2020:08:32:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "149.202.81.xx"
nginx-5976c8f8d6-s674r nginx 10.100.150.255 - - [06/Jun/2020:08:32:18 +0000] "GET / HTTP/1.1" 200 612 "-" "ELB-HealthChecker/2.0" "-"
nginx-5976c8f8d6-s674r nginx 10.100.102.177 - - [06/Jun/2020:08:32:18 +0000] "GET / HTTP/1.1" 200 612 "-" "ELB-HealthChecker/2.0" "-"
nginx-5976c8f8d6-s674r nginx 10.100.129.196 - - [06/Jun/2020:08:32:18 +0000] "GET / HTTP/1.1" 200 612 "-" "ELB-HealthChecker/2.0" "-"
os-virtualretail picture os-virtualretail on 6 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

joseppla picture joseppla  路  5Comments
hieu29791 picture hieu29791  路  4Comments
NickEAVE picture NickEAVE  路  3Comments
benwilson512 picture benwilson512  路  5Comments
brylex418 picture brylex418  路  4Comments