kops-installed kubernetes-iptables-setup.service voids weave-npc rules from working, making kops + weave unusable for kubernetes network policies

[jjo]

1. What kops version are you running? The command kops version, will display
   this information.

Version 1.8.0 (git-5099bc5)

1. What Kubernetes version are you running? kubectl version will print the
   version if a cluster is running or provide the Kubernetes version specified as
   a kops flag.

1.8.5, using image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02
and weave / weave-npc manually upgraded images to 2.1.3
(as there's no kops 1.8 release including https://github.com/kubernetes/kops/pull/3944)

1. What cloud provider are you using?

AWS

1. What commands did you run? What is the simplest way to reproduce this issue?

Deploy a 1.8.5 cluster with 'networking: weave' via kops-1.8

1. What happened after the commands executed?

Expecting some simple network policies example like
https://kubernetes.io/docs/concepts/services-networking/network-policies/#default-deny-all-ingress-traffic
to work

1. What did you expect to happen?

Traffic DROPped as per kubernetes network policy

1. Please provide your cluster manifest.

N/A really - using weave as (and above image):

  networking:
    weave: {}

1. Please run the commands with most verbose logging by adding the -v 10 flag.
   Paste the logs into this report, or in a gist and provide the gist link here.

2. Anything else do we need to know?

kops firewalling setup is added by https://github.com/kubernetes/kops/blob/master/nodeup/pkg/model/firewall.go:

• buildFirewallScript() -> creates a file in the node as /home/kubernetes/bin/iptables-setup
• buildSystemdService() -> creates a systemd one-off kubernetes-iptables-setup.service to run above

This interferes with weave-npc DaemonSet iptables hooks, as weave-npc rules will
be added after above the 3 kops added rules (thanks @mmikulicic for the hard debugging on this!)

somenode:~# iptables-save |egrep -B14 'FORWARD.*WEAVE-NPC$'
:WEAVE-NPC - [0:0]
:WEAVE-NPC-DEFAULT - [0:0]
:WEAVE-NPC-INGRESS - [0:0]
-A INPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kubernetes forward rules" -j KUBE-FORWARD
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -p tcp -j ACCEPT
-A FORWARD -p udp -j ACCEPT
-A FORWARD -p icmp -j ACCEPT
-A FORWARD -o weave -j WEAVE-NPC

[jjo]

BTW below cluster.yaml hook works-around this issue by having pre-set
FORWARD policy to ACCEPT, thus voiding iptables-setup logic
(verified working via node addition):

  hooks:
  - name: iptables-workaround-kops-4345
    roles:
    - Node
    before:
    - kubernetes-iptables-setup.service
    manifest: |
      Type=oneshot
      RemainAfterExit=yes
      ExecStart=/sbin/iptables -P FORWARD ACCEPT

[chrislovecnm]

/cc @bboreham

@jjo what specific rule(s) need to change for weave?

/cc @caseydavenport does this impact calico as well?

[caseydavenport]

@chrislovecnm from the description above, I don't think this will affect Calico.

[jjo]

@jjo what specific rule(s) need to change for weave?

The issue is that (freshly after boot) added iptables -A FOWARD -p <proto> -j ACCEPT
will void any filtering decision by rules added afterwards, as weave-npc does (dunno
about Calico). These rules are added only if FORWARD policy is DROP, my
workaround sets it to ACCEPT before, so that this condition is not met.

We could instead do that in the same script (flip the default policy to ACCEPT),
although arguably it's not exactly the same as (only) ACCEPTing
tcp, udp, icmp, as the former would also accept those other
IP protocols (e.g. IPIP, IPSec/ESP, IGMP to mention some).

In any case, seems that we'd need to revisit the logic behind that
(https://github.com/kubernetes/kubernetes/issues/40182) together
with network policy controllers in place.

[bboreham]

I believe this would be mostly fixed by https://github.com/weaveworks/weave/pull/3210 wherein we insert our drop rule at the top of the chain.

“mostly” because someone can still come along later and insert an accept rule ahead of our drop rule.

[chrislovecnm]

@bboreham that will be in weave 2.2?

[bboreham]

Yes, just released https://github.com/weaveworks/weave/releases/tag/v2.2.0

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[sstarcher]

This is still an issue with 1.9.0 and running
image: weaveworks/weave-npc:2.3.0

[sstarcher]

Looks like with kops 1.9.0 and the hook work around above I'm still seeing it fail to restrict egress using a default egress policy.

apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Egress

[jjo]

@sstarcher the issue reported here has been fixed already (weave >= 2.3),
the issue you're facing is weave not supporting egress at all :(

[sstarcher]

@jjo let me retest, but I was using weave 2.3.0 and it looked like it was still broken.

[sstarcher]

@jjo I just confirmed spinning up a new cluster with kops 1.9.0 and weave installs weave image: weaveworks/weave-npc:2.3.0 and egress is still non-functional. @bboreham

I also tested calico with a brand new cluster and egress was blocked as specified in the network policy.

[bboreham]

Not clear why you are @-ing me. The tracking issue for support of egress policies in Weave Net is https://github.com/weaveworks/weave/issues/2624 as linked above.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[bboreham]

The original issue was fixed in Weave Net 2.2.0

/close