Cert-manager: Add netowrk policy allowance into documentation

[tschirmer]

We've got calico installed on our stack and found configuring this to be a challenge because of the documentation on the solvers. We've added this networking policy (replace {{namespace}} with your project namespace). Would love to see this added to the documentation / improved on to only allow the ports required.

---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-cert-manager-resolver-reverse
  namespace: cert-manager
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/instance: cert-manager
  egress:
    - namespaceSelector:
        matchLabels:
          acme.cert-manager.io/http01-solver: true
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-cert-manager-resolver
  namespace: "{{namespace}}"
spec:
  podSelector:
    matchLabels:
      acme.cert-manager.io/http01-solver: "true"
  ingress:
    - namespaceSelector:
        matchLabels:
          app.kubernetes.io/instance: cert-manager

[GramozKrasniqi]

The first policy has a typo engress->egress ;)

[tschirmer]

@GramozKrasniqi Thanks! Just updated it.

[GramozKrasniqi]

@tschirmer i would also allow port 8089 for the acme-http-solver pod. In case that you have deny_all for a namespace the ingress controller wont be able to route the traffic in there ;)

ingress:
  - ports:
    - port: 8089
      protocol: TCP

[tschirmer]

Hey @GramozKrasniqi

That should be covered by the pod selection shouldn't it?

 podSelector:
    matchLabels:
      acme.cert-manager.io/http01-solver: "true"

That should allow all ports between pods with those labels, not just port 8089. I've got deny all on my namespace and it's working. I could be wrong through.

[GramozKrasniqi]

@tschirmer i think you are defining communication routes only from/to cert-manager components to acme-solver and vice versa.

When configuring HTTP-01 challenges, internally an Ingress object will be created pointing to a service (acme-solver service also auto created) pointing to pods (acme-solver pods also auto created). That is why we have to allow ingress to acme-solver pod also from the Ingress Component (be it Application Gateway instance, Nginx etc) or just open ingress to pod 8089 for everyone.

This is from Lets Encrypt Page:
This is the most common challenge type today. Let’s Encrypt gives a token to your ACME client, and your ACME client puts a file on your web server at http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN>. That file contains the token, plus a thumbprint of your account key. Once your ACME client tells Let’s Encrypt that the file is ready, Let’s Encrypt tries retrieving it (potentially multiple times from multiple vantage points). If our validation checks get the right responses from your web server, the validation is considered successful and you can go on to issue your certificate. If the validation checks fail, you’ll have to try again with a new certificate.

[sc250024]

I have something else to add: the webhook itself _also_ needs whitelisting. I currently have the following NetworkPolicy in my cert-manager namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ingress-allow-same-namespace
  namespace: cert-manager
spec:
  ingress:
  - from:
    - podSelector: {}
  podSelector: {}
  policyTypes:
  - Ingress

Which allows communication between pods in the namespace. However, when applying a new certificate, I ran into the following issue (the output is kubectl -v 10 when applying the new certificate):

I0211 14:34:22.738258   48439 request.go:1017] Request Body: {"apiVersion":"cert-manager.io/v1alpha2","kind":"Certificate","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"cert-manager.io/v1alpha2\",\"kind\":\"Certificate\",\"metadata\":{\"annotations\":{\"reflector.v1.k8s.emberstack.com/secret-reflection-allowed\":\"true\",\"reflector.v1.k8s.emberstack.com/secret-reflection-auto-enabled\":\"true\"},\"name\":\"app-eu-stackstate-com\",\"namespace\":\"cert-manager\"},\"spec\":{\"commonName\":\"*.mycompany.com\",\"dnsNames\":[\"*.mycompany.com\",\"mycompany.com\"],\"issuerRef\":{\"kind\":\"ClusterIssuer\",\"name\":\"le-prod\"},\"keyAlgorithm\":\"ecdsa\",\"keyEncoding\":\"pkcs8\",\"keySize\":384,\"secretName\":\"star-mycompany-com-tls\"}}\n","reflector.v1.k8s.emberstack.com/secret-reflection-allowed":"true","reflector.v1.k8s.emberstack.com/secret-reflection-auto-enabled":"true"},"name":"app-eu-stackstate-com","namespace":"cert-manager"},"spec":{"commonName":"*.mycompany.com","dnsNames":["*.mycompany.com","mycompany.com"],"issuerRef":{"kind":"ClusterIssuer","name":"le-prod"},"keyAlgorithm":"ecdsa","keyEncoding":"pkcs8","keySize":384,"secretName":"star-mycompany-com-tls"}}
I0211 14:34:22.738476   48439 round_trippers.go:423] curl -k -v -XPOST  -H "Content-Type: application/json" -H "User-Agent: kubectl/v1.17.2 (darwin/amd64) kubernetes/59603c6" -H "Accept: application/json" 'https://<some-eks-endpoint>.gr7.eu-west-1.eks.amazonaws.com/apis/cert-manager.io/v1alpha2/namespaces/cert-manager/certificates'
I0211 14:34:52.823007   48439 round_trippers.go:443] POST https://<some-eks-endpoint>.gr7.eu-west-1.eks.amazonaws.com/apis/cert-manager.io/v1alpha2/namespaces/cert-manager/certificates 500 Internal Server Error in 30084 milliseconds
I0211 14:34:52.823066   48439 round_trippers.go:449] Response Headers:
I0211 14:34:52.823125   48439 round_trippers.go:452]     Audit-Id: <id>
I0211 14:34:52.823136   48439 round_trippers.go:452]     Content-Type: application/json
I0211 14:34:52.823144   48439 round_trippers.go:452]     Content-Length: 643
I0211 14:34:52.823151   48439 round_trippers.go:452]     Date: Tue, 11 Feb 2020 13:34:52 GMT
I0211 14:34:52.823222   48439 request.go:1017] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"webhook.cert-manager.io\": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}]},"code":500}
I0211 14:34:52.823714   48439 helpers.go:203] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "error when creating \"/path/to/some-new-certificate.yaml\": Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)",
  "reason": "InternalError",
  "details": {
    "causes": [
      {
        "message": "failed calling webhook \"webhook.cert-manager.io\": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
      }
    ]
  },
  "code": 500
}]
F0211 14:34:52.823817   48439 helpers.go:114] Error from server (InternalError): error when creating "/path/to/some-new-certificate.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

In my case, removing my GlobalNetworkPolicy from Calico, and the NetworkPolicy above found in the cert-manager namespace, and reapplying actually did work.

I'm a little confused by this because I'm not sure which part of Kubernetes is trying to contact https://cert-manager-webhook.cert-manager.svc:443. What extra ingress rule should be allowed for the webhook, and from where?

[GramozKrasniqi]

@sc250024 the webhook is called directly from the k8s api server.

Please check the following documentation:
https://docs.cert-manager.io/en/release-0.9/getting-started/webhook.html

[sc250024]

@GramozKrasniqi Thank you for the response, that does make sense. If I'm implementing a NetworkPolicy on a managed platform like Amazon EKS, what podSelector or ipBlock would we whitelist though?

[GramozKrasniqi]

Hey @sc250024 you dont have to provide any of that.

Cloud providers give a managed instance of k8s master node and provide high availability by putting the master nodes in different availability zones. Its hard to find a continious ip blocks for that.

In that case you can just allow ingress traffit to port 6443 in the network policy (which by default is used by this cert-manager webhook) without specifying podselector or ipselector.

Please also check out if you need to open any inbound firewall rules on port 6443 on the virtual network (depends on the cloud provider).

[munnerz]

Would someone be able to put together some information to go into the docs on this? I think a page under the FAQ section may fit? Although if you see anywhere else that it'd be more suitable then go for it 😄 https://cert-manager.io/docs/faq/

/kind documentation
/area deploy
/priority backlog
/good-first-issue
/help

[jetstack-bot]

@munnerz:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

Would someone be able to put together some information to go into the docs on this? I think a page under the FAQ section may fit? Although if you see anywhere else that it'd be more suitable then go for it 😄 https://cert-manager.io/docs/faq/

/kind documentation
/area deploy
/priority backlog
/good-first-issue
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sc250024]

Would someone be able to put together some information to go into the docs on this? I think a page under the FAQ section may fit? Although if you see anywhere else that it'd be more suitable then go for it 😄 cert-manager.io/docs/faq

/kind documentation
/area deploy
/priority backlog
/good-first-issue
/help

Sure. I still didn't get it to work in the way @GramozKrasniqi described. The reason is very clear: if you're using EKS, there are no master nodes, so you can't specify a source in your networking rules (using podSelector and nameSpace selector) because there's nothing to select.

@GramozKrasniqi you're going to have to explicitly post how you accomplished that, if you could please.