Contour: Internal and External ingress with 2 contour deployments

[so0k]

Objective

I need to configure internal and external ingress controllers on AWS with NLB

Internal Ingress is served by an internal NLB (only accessible from within the VPC)
External Ingress is served by an external NLB (publicly accessible)

What steps did you take and what happened:

I'm using deployment/ds-hostnet-split as a guideline.

full final manifest

• I'm using the same rbac / crb / sa for both contour deployments
• I have 2 Contour deployments:
  
  
  
  • name & label:contour-external for deploy / svc (clusterIP) manifest
    
    the arguments:
    
    <br /> <ul><br /> <li>serve</li><br /> <li>--incluster</li><br /> <li>--xds-address</li><br /> <li>0.0.0.0</li><br /> <li>--xds-port</li><br /> <li>$(CONTOUR_EXTERNAL_SERVICE_PORT)</li><br /> <li>--envoy-http-port</li><br /> <li>"80" #&lt;&lt;&lt;&lt;&lt;</li><br /> <li>--envoy-https-port</li><br /> <li>"443" #&lt;&lt;&lt;&lt;&lt;</li><br /> <li>--ingress-class-name=envoy-external<br /><br /> ````</li><br /> </ul></li> <li>name &amp; label: <code>md5-9522bfa90e8181d32bb1a6b8880dbbbd</code> for deploy / svc (clusterIP) manifest<br /> the arguments:<br />
    
    
    
    
    
    • serve
    
    
    
    • --incluster
    
    
    
    • --xds-address
    
    
    
    • 0.0.0.0
    
    
    
    • --xds-port
    
    
    
    • $(CONTOUR_INTERNAL_SERVICE_PORT)
    
    
    
    • --envoy-http-port
    
    
    
    • "8080" #<<<<<
    
    
    
    • --envoy-https-port
    
    
    
    • "8443" #<<<<<
    
    
    
    • --ingress-class-name=envoy-internal
      
      
      ````
    
    
    
  
  
• I have 2 Envoy daemonsets with hostNetwork: true:
  
  
  
  • name & label: envoy-external for ds / svc (LoadBalancer:NLB) manifest
    
    environment:
    
    <ul> <li>name: NODE_NAME<br /> valueFrom:<br /> fieldRef:<br /> fieldPath: spec.nodeName<br /> <code>md5-aa3aacb02e8e289919534fc97a4c017c</code></li> <li>-c</li> <li>/config/contour.yaml</li> <li>--service-cluster</li> <li>envoy-external-cluster</li> <li>--service-node</li> <li>$(NODE_NAME)<br /> `<code>md5-afe2f0e2d5f52707501a9b5b04223cd5</code></li> <li>--xds-address</li> <li>$(CONTOUR_EXTERNAL_SERVICE_HOST)</li> <li>--xds-port</li> <li>$(CONTOUR_EXTERNAL_SERVICE_PORT)</li> <li>--stats-port</li> <li>"8002"</li> <li>--admin-port</li> <li>"9001"<br /> HealthCheck uses8002 preStop uses9001</li><br /> </ul></li> <li>name &amp; label: <code>md5-ec37671cba9237d68f0c16d7cf270d00</code> for ds / svc (LoadBalancer:NLB) manifest<br /> environment is the same as above, using fieldRef for spec.NodeName<br /> args:<br /> <code>md5-f7fada03ba973834fca4bfd9379c6490</code><code>md5-c2a8520be8f14be728ce14f7c1a554ad</code>8004<code>md5-454b2eae6e3e3630738bde3deee74728</code>9002
  
  
  

What did you expect to happen:

I expect both envoy ds to run side by side, but they exit(1) without any logs, I verified that they do not use the same hostPorts and I reviewed and validated the configuration created by contour bootstrap..
If I delete one ds the other starts without issue and vice versa.

SSH to the node and running a docker container without host network with the same envoy allows both containers to run side by side.

Anything else you would like to add:

here is a gist for full manifests:

https://gist.github.com/so0k/2cb93c405cf05fcb8ecb73fa665efbd0

Environment:

• Contour version: 0.8.1
• Kubernetes version: 1.11.6
• Kubernetes installer & version: kops 1.11
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): CoreOS

[so0k]

without using hostNetwork - this works: new-manifest - but the clientIP is not preserved

[jbeda]

Ok -- here are the ports I see in the original manifest

• service/envoy-external
  
  
  
  • 80->80
  
  
  • 443->443
  
  
• service/contour-external - 8001->8001
• deployment/contour-external (hostNetwork: false)
  
  
  
  • xds - CONTOUR_EXTERNAL_SERVICE_PORT = 8001
  
  
  • envoy-http - 80
  
  
  • envoy-https - 443
  
  
  • debug - 8000 (only specified in ports section, not on command line)
  
  
• daemonset/envoy-external (hostNetwork: true)
  
  
  
  • http: 80
  
  
  • https: 443
  
  
  • stats: 8002
  
  
  • admin: 9001
  
  
• service/envoy-internal
  
  
  
  • 80->8080
  
  
  • 443->8443
  
  
• service/contur-internal
  
  
  
  • 8001->8001
  
  
• deployment/contour-internal (hostNetwork: false)
  
  
  
  • xds: CONTOUR_INTERNAL_SERVICE_PORT = 8001
  
  
  • envoy http: 8080
  
  
  • envoy https: 8443
  
  
  • debug - 8000 (only specified in ports section, not on command line)
  
  
• daemonset/envoy-internal (hostNetwork: true)
  
  
  
  • http: 8080
  
  
  • https: 8443
  
  
  • stats: 8004
  
  
  • admin: 9002
  
  

From first blush, it looks like there are no overlapping ports on the things that are running as host networking and other ports look to be wired up fine.

@davecheney -- do you know if envoy is opening any other ports that aren't specified? Is there a way to make it dump more verbose logging to debug why it is exiting?

@so0k -- might be useful to ssh into the node and use lsof to find what ports the envoy is listening on to see if there is anything unexpected. I don't know the syntax off the top of my head but it looks like some quick googling leads to some sane guides.

[so0k]

running netstat -nltp and playing around the ds to see what ports envoy-external vs envoy-internal each open on the host did not reveal any overlapping ports.

I will try this again tomorrow with a fresh mind as I spent quite a lot of time breaking my head on this already

[stevesloka]

Hey @so0k can you try adding this argument to one of the Envoy daemonset manifests?

- --base-id
- "2"

ref: https://www.envoyproxy.io/docs/envoy/v1.8.0/operations/cli.html#cmdoption-base-id

[so0k]

great find! Adding base-id: new gist - didn't have time to test it yet

[anapsix]

@stevesloka / @so0k, confirmed <3!
Works as advertised (though still having problems with proxy-proto described in https://github.com/heptio/contour/issues/793)

[davecheney]

@anapsix can you please raise a new issue to talk about your problems using proxy-protocol. Thank you.

@so0k what is required to close this issue?

[so0k]

@davecheney - should I do a PR with documentation about adding --base-id flag to envoy if running multiple ds in hostNetwork ?

[davecheney]

Documentation issue, I’m assigning this to beta.1. @so0k feel free to send a docs PR before then.

[davecheney]

@so0k ping

[jpeach]

Here's a working configuration that uses kustomize to deploy internal and external Contours:

https://github.com/jpeach/kustomize/tree/master/knative/configurations/contour

[so0k]

Sorry, I moved company and I'm afraid I won't be able to finish this documentation task until I have a use case to switch ingress over to Contour

[davecheney]

That’s ok, all the best with your new endeavour.

On 17 Feb 2020, at 14:32, so0k [e-mail] wrote:



Sorry, I moved company and I'm afraid I won't be able to finish this documentation task

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fprojectcontour%2Fcontour%2Fissues%2F855%3Femail_source%3Dnotifications%26email_token%3DAAABYA3UCMFK4R3EK7RTSJDRDIANTA5CNFSM4GRUCJK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL46PIY%23issuecomment-586803107&data=02%7C01%7Ccheneyd%40vmware.com%7Cb7aed1f560834a08fddc08d7b35a0b80%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637175071633191550&sdata=adKr222vy0TXpR6e7lQA%2FcNtIqkHUMk7b5Ltg9hJA6Q%3D&reserved=0, or unsubscribehttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAABYAZ3AI774KCZDOAAN5LRDIANTANCNFSM4GRUCJKQ&data=02%7C01%7Ccheneyd%40vmware.com%7Cb7aed1f560834a08fddc08d7b35a0b80%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637175071633191550&sdata=5p%2BKFFkWVeCA0bBezq5FSBWaWh4%2BEIBm6xztrv9gPM8%3D&reserved=0.