Istio: GateWay Ssl Error

Created on 7 Jun 2018  Â·  59Comments  Â·  Source: istio/istio 
Version: 0.8.0
GitRevision: 6f9f420f0c7119ff4fa6a1966a6f6d89b1b4db84

I have this error when i try to reach a pods behind a Gateway (via https)

curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 104.199.103.101:443

This may be related to this warning in the logs of the gateway pods

[2018-06-07 15:27:38.480][17][warning][config] external/envoy/source/server/listener_manager_impl.cc:254] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)

Any ideas?

Thanks

arenetworking communithelp wanted kinneed more info
Source
picture hamon-e
👍3

Most helpful comment

Got exact the same issue:

SSL Certificates are in place:
HTTP flow works fine

But! once I provision both Gateways - HTTP and HTTPS. SSL connection begin to work
removing HTTP leads to same error
(35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to mydomain.com

picture eigokor on 23 Jul 2018
👍6

All 59 comments

Could you please send full pilot and gateway pod logs?

andraxylia picture andraxylia on 13 Jun 2018

same to me

ghost picture ghost on 11 Jul 2018

@vadimeisenbergibm Does this look like it may be the same problem that you and @dcberg were investigating yesterday? If so, any resolution?

@hamon-e and @wjwinner Can you provide more details?

frankbu picture frankbu on 11 Jul 2018

@frankbu no, the issue was VPN/firewall on the client side. More information is required. See also https://istio.io/docs/tasks/traffic-management/secure-ingress/#troubleshooting

vadimeisenbergibm picture vadimeisenbergibm on 11 Jul 2018

I am having this issue specifically when using a AWS NLB with a Istio Gateway on HTTPS.

From istio-ingressgateway logs:

adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)

The gateway and virtualservice are not working under that scenario (requests are not routed). Notice it does work with the following combinations:

  • AWS ELB (classic) and HTTPS
  • AWS NLB and HTTP

From istio-ingressgateway k8s service definition (notice the annotations to set up the NLB):

apiVersion: v1
kind: Service
metadata:
  name: istio-ingressgateway
  namespace: istio-system    
  labels:
    chart: ingressgateway-0.8.0
    release: istio
    heritage: Tiller
    istio: ingressgateway
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
spec:
  type: LoadBalancer
  selector:
    istio: ingressgateway
  ports:
    -
      name: http
      nodePort: 31380
      port: 80
    -
      name: https
      nodePort: 31390
      port: 443
    -
      name: tcp
      nodePort: 31400
      port: 31400

Running Istio 0.8.0 - May 2018 Release (LTS) and Kubernetes 1.11.0.

juliaaano picture juliaaano on 17 Jul 2018

@juliaaano The warning is probably insignificant. Note the troubleshooting section in https://preliminary.istio.io/docs/tasks/traffic-management/secure-ingress/#troubleshooting. Could you please perform the step 1 and share the output? And also the command that you use to access your ingress?

vadimeisenbergibm picture vadimeisenbergibm on 17 Jul 2018

@vadimeisenbergibm The step 1 suggests the following:

kubectl get svc -n istio-system
$ echo INGRESS_HOST=$INGRESS_HOST, SECURE_INGRESS_PORT=$SECURE_INGRESS_PORT

I don't get it how sharing with you my ingress host will be of any help. I got this working with NLB and HTTP and I also got it working with ELB and HTTPS. It's the combination of NLB and HTTPS that is giving me the aforementioned warning and causing the gateway to not route the requests.

The result of that command above will be a AWS load balancer dns name, such as ae96ead38897911e8adab0a3d68bae34-2142157781.ap-southeast-1.elb.amazonaws.com and the port is 443

Thanks in advance for putting some time and sorry if I have misunderstood something.

juliaaano picture juliaaano on 18 Jul 2018

@juliaaano For ELB and HTTPS, you get the same warning, don't you? What is different in your configuration between ELB/HTTPS and NLB/HTTPS, only the annotations? Are the VirtualService and Gateway definitions the same?

vadimeisenbergibm picture vadimeisenbergibm on 18 Jul 2018

I'm also experiencing a similar issue.

In my case using an NLB in combination with HTTPS (I've tried an ELB as well with no better luck).

What I see is:

curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 18.210.31.22:443

and a very similar error is also thrown if I try to connect internally to the ClusterIp of the service:

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 10.100.243.178:443

Certificates have been created as suggested using https://github.com/nicholasjackson/mtls-go-example

This is the result of the suggested troubleshooting steps:

  1. Host would be an NLB address, port 443
    2.
$ kubectl exec -it -n istio-system $(kubectl -n istio-system get pods -l istio=ingressgateway -o jsonpath='{.items[0].metadata.name}') -- ls -al /etc/istio/ingressgateway-certs

total 8
drwxrwxrwt 3 root root   80 Jul 19 08:41 .
drwxr-xr-x 1 root root 4096 Jul 19 08:41 ..
drwxr-xr-x 2 root root   40 Jul 19 08:41 ..2018_07_19_08_41_16.761205109
lrwxrwxrwx 1 root root   31 Jul 19 08:41 ..data -> ..2018_07_19_08_41_16.761205109
  1. Logs from the ingress gateway show:
[2018-07-19 08:59:53.102][52][warning][config] external/envoy/source/server/listener_manager_impl.cc:254] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)
[2018-07-19 08:59:53.103][52][info][upstream] external/envoy/source/server/lds_api.cc:62] lds: add/update listener '0.0.0.0_443'
[2018-07-19 08:59:58.255][52][warning][config] external/envoy/source/server/listener_manager_impl.cc:254] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)
[2018-07-19 08:59:58.256][52][info][upstream] external/envoy/source/server/lds_api.cc:62] lds: add/update listener '0.0.0.0_443'

Thanks!

claudio-benfatto picture claudio-benfatto on 19 Jul 2018
👍1

@claudio-benfatto Note that you do not have the key and the certificate in /etc/istio/ingressgateway-certs:

https://preliminary.istio.io/docs/tasks/traffic-management/secure-ingress/#troubleshooting

tls.crt and tls.key should exist in the directory contents.

vadimeisenbergibm picture vadimeisenbergibm on 19 Jul 2018

@vadimeisenbergibm really sorry, it somehow got cut off the copy-paste:

~ $ kubectl exec -it -n istio-system $(kubectl -n istio-system get pods -l istio=ingressgateway -o jsonpath='{.items[0].metadata.name}') -- ls -al /etc/istio/ingressgateway-certs

total 8
drwxrwxrwt 3 root root  120 Jul 19 14:37 .
drwxr-xr-x 1 root root 4096 Jul 19 14:27 ..
drwxr-xr-x 2 root root   80 Jul 19 14:37 ..2018_07_19_14_37_13.238515031
lrwxrwxrwx 1 root root   31 Jul 19 14:37 ..data -> ..2018_07_19_14_37_13.238515031
lrwxrwxrwx 1 root root   14 Jul 19 14:37 tls.crt -> ..data/tls.crt
lrwxrwxrwx 1 root root   14 Jul 19 14:37 tls.key -> ..data/tls.key
claudio-benfatto picture claudio-benfatto on 19 Jul 2018

@jsenon Did you have AWS with NLB working with Istio Ingress Gateway https://preliminary.istio.io/docs/tasks/traffic-management/secure-ingress ?

vadimeisenbergibm picture vadimeisenbergibm on 19 Jul 2018

@vadimeisenbergibm I can check that part

jsenon picture jsenon on 19 Jul 2018

@vadimeisenbergibm I've just finish to test without SSL and it's works with booking example.

Gateway:

Name:         bookinfo-gateway
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  networking.istio.io/v1alpha3
Kind:         Gateway
Metadata:
  Creation Timestamp:  2018-07-20T22:08:26Z
  Generation:          1
  Resource Version:    10640
  Self Link:           /apis/networking.istio.io/v1alpha3/namespaces/default/gateways/bookinfo-gateway
  UID:                 6cd5d276-8c69-11e8-a24d-02c2dbb86c04
Spec:
  Selector:
    Istio:  ingressgateway
  Servers:
    Hosts:
      *
    Port:
      Name:      http
      Number:    80
      Protocol:  HTTP
Events:          <none>

Svc : 

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  labels:
    chart: ingressgateway-0.8.0
    heritage: Tiller
    istio: ingressgateway
    release: istio
  name: istio-ingressgateway
  namespace: istio-system
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    nodePort: 31380
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    nodePort: 31390
    port: 443
    protocol: TCP
    targetPort: 443
  - name: tcp
    nodePort: 31400
    port: 31400
    protocol: TCP
    targetPort: 31400
  selector:
    istio: ingressgateway
  sessionAffinity: None
  type: LoadBalancer

istio-ingressgateway       LoadBalancer   100.70.244.66    af90bcaXXXXX24d02c2dbb86c0-ec474XXX0.elb.eu-central-1.amazonaws.com   80:31380/TCP,443:31390/TCP,31400:31400/TCP                            9m        istio=ingressgateway

$ curl -s -o /dev/null -w "%{http_code}"  http://af90bcaXXXXX24d02c2dbb86c0-ec474XXX0.elb.eu-central-1.amazonaws.com/productpage                                                                
200%

Hope this will help

jsenon picture jsenon on 21 Jul 2018

@jsenon Thanks, and what about SSL?

vadimeisenbergibm picture vadimeisenbergibm on 21 Jul 2018

The situation I hit with this scenario was that the TLS cert config was not properly stored with the Gateway. You can use the istioctl to validate that the Gateway config has the TLS config. This is fixed in the latest prelim build.

The other part is that your load balancer implementation must pass the expected SNI that matches the CN in the cert. If it does not then you will get a handshake error.

dcberg picture dcberg on 21 Jul 2018

Got exact the same issue:

SSL Certificates are in place:
HTTP flow works fine

But! once I provision both Gateways - HTTP and HTTPS. SSL connection begin to work
removing HTTP leads to same error
(35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to mydomain.com

eigokor picture eigokor on 23 Jul 2018
👍6

@eigokor is it happening on AWS with NLB?

vadimeisenbergibm picture vadimeisenbergibm on 23 Jul 2018

@vadimeisenbergibm AWS, ELB

eigokor picture eigokor on 23 Jul 2018

@andraxylia @linsun What is our policy regarding Istio on AWS? Should I debug it? Are there any members in the community who could work on it?

vadimeisenbergibm picture vadimeisenbergibm on 23 Jul 2018

@vadimeisenbergibm I do not know who can help with AWS specific issues, so I added the help wanted label.

@loewenstein @christian-posta any idea with this AWS specific setup?

andraxylia picture andraxylia on 23 Jul 2018

Does the nlb and more precisely target still healthy ?
Does the security group on the node have correct filtering (normally applied by k8s lb creation) ?

On 23 Jul 2018, at 7:23 PM, Andra Cismaru notifications@github.com wrote:

@vadimeisenbergibm I do not know who can help with AWS specific issues, so I added the help wanted label.

@loewenstein @christian-posta any idea with this AWS specific setup?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

jsenon picture jsenon on 23 Jul 2018

Has mention by @vadimeisenbergibm is it classical load balancer, network load balancer or application load balancer ?

jsenon picture jsenon on 23 Jul 2018

it is ELB - Classic Load Balancer, created by k8s service by default
mean time will keep both 80 and 443 in Gateway configuration.
redirect to SSL works for me

  - port:
      number: 80
      name: http
      protocol: HTTP
    tls:
      httpsRedirect: true
eigokor picture eigokor on 23 Jul 2018

Hello, I've tested e2e solution with aws nlb

  1. I've follow https://istio.io/docs/tasks/traffic-management/ingress/ to deploy httpbin with http firstly

  2. I've follow https://istio.io/docs/tasks/traffic-management/secure-ingress/ for https implementation

  3. I've adapt a new gateway to handle 443 AND 80

```cat < apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: httpbin-gateway
spec:
selector:
istio: ingressgateway # use istio default ingress gateway
servers:

  • port:
    number: 443
    name: https
    protocol: HTTPS
    tls:
    mode: SIMPLE
    serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
    privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:

    • "httpbinssl.xxx"

  • port:
    number: 80
    name: http
    protocol: HTTP
    hosts:

    • "httpbinssl.istio.xxx"

      EOF


https is working find (expect auto signed certificate warning):

curl -vv https://httpbinssl.xxx/status/200

  • Trying 18.18xxx...
  • TCP_NODELAY set
  • Connected to httpbinssl.xxx (18.18xxx) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/cert.pem
    CApath: none
  • TLSv1.2 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (OUT), TLS alert, Server hello (2):
  • SSL certificate problem: unable to get local issuer certificate
  • stopped the pause stream!
  • Closing connection 0
    curl: (60) SSL certificate problem: unable to get local issuer certificate
    More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.


And http still working:

curl -vv http://httpbinssl.xxx/status/200

  • Trying 18.18xxx...
  • TCP_NODELAY set
  • Connected to httpbinssl.xxx (18.18xxx) port 80 (#0)

GET /status/200 HTTP/1.1
Host: httpbinssl.xxx
User-Agent: curl/7.54.0
Accept: /

< HTTP/1.1 200 OK
< server: envoy
< date: Mon, 23 Jul 2018 19:58:15 GMT
< content-type: text/html; charset=utf-8
< access-control-allow-origin: *
< access-control-allow-credentials: true
< content-length: 0
< x-envoy-upstream-service-time: 4
<

  • Connection #0 to host httpbinssl.xxx left intact
    ```
jsenon picture jsenon on 23 Jul 2018

so it does look like we cannot have only 443 HTTPS port configured on Gateway side, and 80 HTTP need to be there anyway

eigokor picture eigokor on 23 Jul 2018

No you can have only https port let perform test and post output

jsenon picture jsenon on 23 Jul 2018

Gateway ssl:

cat <<EOF | istioctl create -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gateway
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - "httpbinssl.xxx"                                                                                                                                                                                                                                                                         EOF

Created config gateway/default/httpbin-gateway at revision 439033

curl -vv http://httpbinssl.xxx/status/200                                                                                                                                                                                                
*   Trying 18.18xxx...
* TCP_NODELAY set
* Connection failed
* connect to 18.18xxx port 80 failed: Connection refused
* Failed to connect to httpbinssl.xxx port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to httpbinssl.xxx port 80: Connection refused

curl -vv https://httpbinssl.xxx/status/200                                                                                                                                                                                                
*   Trying 18.18xxx...
* TCP_NODELAY set
* Connected to httpbinssl.xxx (18.18xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.
jsenon picture jsenon on 23 Jul 2018

hum you want to perform redirect?

jsenon picture jsenon on 23 Jul 2018 
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: nginx-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - "nginx-test.xxx.xxx"

→ curl -vv -L https://nginx-test.xxx.xxx:443
* Rebuilt URL to: https://nginx-test.xxx.xxx:443/
*   Trying xx.xx.xx.xx...
* TCP_NODELAY set
* Connected to nginx-test.xxx.xxx:443 (xx.xx.xx.xx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to nginx-test.xxx.xxx:443
* stopped the pause stream!
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to nginx-test.xxx.xxx:443

→ openssl s_client -showcerts -servername nginx-test.xxx.xxx -connect nginx-test.xxx.xxx:443
CONNECTED(00000006)
140736284562376:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.50.2/libressl/ssl/s23_lib.c:124:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 359 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
---



md5-edb20d25773d0fa589a8bf3b044b7d48



apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: nginx-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - "nginx-test.xxx.xxx"
eigokor picture eigokor on 23 Jul 2018

Seems related to cert, not gateway or virtual service. How do you have created and imported certs?

jsenon picture jsenon on 23 Jul 2018

cert-manager, signed with lets`encrypt

but if it is cert issue why SSL starts working once HTTP configured?

→ curl -vv -L https://nginx-test.xxx.xxx
* Rebuilt URL to: https://nginx-test.xxx.xxx/
*   Trying xxx.xxx.xxx.xxx...
* TCP_NODELAY set
* Connected to nginx-test.xxx.xxx (xxx.xxx.xxx.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.xxx.xxx
*  start date: Jul 23 10:26:04 2018 GMT
*  expire date: Oct 21 10:26:04 2018 GMT
*  subjectAltName: host "nginx-test.xxx.xxx" matched cert's "*.xxx.xxx"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7ff0a8808400)
> GET / HTTP/2
> Host: nginx-test.xxx.xxx
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< server: envoy
< date: Mon, 23 Jul 2018 20:30:53 GMT
< content-type: text/html
< content-length: 8
< last-modified: Mon, 23 Jul 2018 07:40:12 GMT
< etag: "5b55865c-8"
< accept-ranges: bytes
< x-envoy-upstream-service-time: 2
<
NGINX-1
* Connection #0 to host nginx-test.xxx.xxx left intact
eigokor picture eigokor on 23 Jul 2018

could you try with https://github.com/nicholasjackson/mtls-go-example?
In the meantime, I ve setup istio with lets encrypt, let me check conf.
Are you using cert-manager?

jsenon picture jsenon on 23 Jul 2018

could you try with auto signed cert linked in my previous post ? you don't have older ingress defined?

jsenon picture jsenon on 23 Jul 2018

I am having essentially the same issue, but I do have an older kubernetes ingress defined mostly used to get a dns record using external-dns

Name:             schema-registry
Namespace:        dev
Address:          
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     *     default-http-backend:80 (<none>)
Annotations:
  external-dns.alpha.kubernetes.io/target:           ingress.dev.comapany.com

would this somehow impact it? When I add 80 to the Gateway it also starts working.

Whats also funny the ELB shows NotInService if only 443 is specified.

Arttii picture Arttii on 30 Jul 2018

Kubernetes 1.10.3 deployed using Kops on AWS
Istio 1.0

Experiencing same issue running through https://istio.io/docs/tasks/traffic-management/secure-ingress/ tutorial - using ELB not NLB.

I find that if I create the Gateway as HTTPS only then curl fails with the SSL_ERROR_SYSCALL immediately after the client hello. However, with HTTP route in Gateway then all is well.

Experienced same issue with both SIMPLE and MUTUAL TLS.

Potentially relevant record from ingressgateway

[2018-08-06 13:04:20.325][16][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:240] gRPC config stream closed: 2, RDS: Failed to generate route http.80 for node &{Kubernetes router 10.1.1.222 istio-ingressgateway-7fd966b5bd-p9dq7.istio-system istio-system.svc.cluster.local map[ISTIO_PROXY_SHA:istio-proxy:6166ae7ebac7f630206b2fe4e6767516bf198313 ISTIO_VERSION:1.0.0 POD_NAME:istio-ingressgateway-7fd966b5bd-p9dq7 istio:sidecar ISTIO_PROXY_VERSION:1.0.0]}: buildGatewayRoutes: could not find server for routeName http.80, have map[https.443.https:[port:<number:443 protocol:"HTTPS" name:"https" > hosts:"httpbin.example.com" tls:<mode:SIMPLE server_certificate:"/etc/istio/ingressgateway-certs/tls.crt" private_key:"/etc/istio/ingressgateway-certs/tls.key" > ]]

Also get previously mentioned error:

[2018-08-06 13:08:03.287][16][warning][config] external/envoy/source/server/listener_manager_impl.cc:257] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)
subnova picture subnova on 6 Aug 2018

I still experience this issue as well (istio version 1.0.0)... (I had performed basic troubleshooting above in this thread)

ingressgateway service

istio-ingressgateway       LoadBalancer   10.100.224.41    <REDACTED>...   80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:31337/TCP,8060:30341/TCP,15030:30834/TCP,15031:30600/TCP   19h

ingressgateway pod details:

kubectl describe pods/istio-ingressgateway-67995c486c-w2pcc -n istio-system
Name:           istio-ingressgateway-67995c486c-w2pcc
Namespace:      istio-system
Node:           ip-172-27-22-150.ec2.internal/172.27.22.150

(OK) Attempt 1: try to connect from a worker to the ingressgateway ClusterIp:

curl -k -vvvv --resolve httpbin.example.com:443:10.100.224.41  https://httpbin.example.com/status/200
<....>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< server: envoy
< date: Mon, 13 Aug 2018 08:23:10 GMT
< content-type: text/html; charset=utf-8
< access-control-allow-origin: *
< access-control-allow-credentials: true
< content-length: 0
< x-envoy-upstream-service-time: 5

(OK)Attempt 2: trying to connect directly to the host port (over https):

curl -k -vvvv --header 'Host: httpbin.example.com' --resolve httpbin.example.com:31390:172.27.12.33  https://httpbin.example.com:31390/status/200

* Added httpbin.example.com:31390:172.27.12.33 to DNS cache
* Hostname httpbin.example.com was found in DNS cache
*   Trying 172.27.12.33...
* TCP_NODELAY set
* Connected to httpbin.example.com (172.27.12.33) port 31390 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=Denial; L=Springfield; O=Dis; CN=*.example.com
*  start date: Aug 10 12:16:35 2018 GMT
*  expire date: Aug 20 12:16:35 2019 GMT
*  issuer: C=US; ST=Denial; O=Dis; CN=*.example.com
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5631c572b8e0)
> GET /status/200 HTTP/2
> Host: httpbin.example.com
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< server: envoy
< date: Mon, 13 Aug 2018 08:45:41 GMT
< content-type: text/html; charset=utf-8
< access-control-allow-origin: *
< access-control-allow-credentials: true
< content-length: 0
< x-envoy-upstream-service-time: 5
<
* Connection #0 to host httpbin.example.com left intact

(OK) Attempt 3: connecting via the ELB (over HTTP):

curl -v http://httpbin.example.com/status/200
*   Trying <REDACTED>...
* TCP_NODELAY set
* Connected to httpbin.example.com (<REDACTED>) port 80 (#0)
> GET /status/200 HTTP/1.1
> Host: httpbin.example.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< server: envoy
< date: Mon, 13 Aug 2018 08:52:07 GMT
< content-type: text/html; charset=utf-8
< access-control-allow-origin: *
< access-control-allow-credentials: true
< content-length: 0
< x-envoy-upstream-service-time: 5
<
* Connection #0 to host httpbin.example.com left intact

(NOT OK) Attempt 4: connecting via the ELB (over HTTPS):

$ curl -k  -vvvv  https://httpbin.example.com/status/200
*   Trying <REDACTED>...
* TCP_NODELAY set
* Connected to httpbin.example.com (<REDACTED>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.example.com
*  start date: Dec 22 00:00:00 2017 GMT
*  expire date: Jan 22 12:00:00 2019 GMT
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET /status/200 HTTP/1.1
> Host: httpbin.example.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 408 REQUEST_TIMEOUT
< Content-Length:0
< Connection: Close
<
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):

and finally my ELB configuration, created automatically by kubernetes LoadBalancer Service:

{
    "LoadBalancerDescriptions": [
        {
            "LoadBalancerName": "<REDACTED>",
            "DNSName": "<REDACTED>.us-east-1.elb.amazonaws.com",
            "CanonicalHostedZoneName": "<REDACTED>.us-east-1.elb.amazonaws.com",
            "CanonicalHostedZoneNameID": "Z35SXDOTRQ7X7K",
            "ListenerDescriptions": [
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 80,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 31380
                    },
                    "PolicyNames": []
                },
                {
                    "Listener": {
                        "Protocol": "HTTPS",
                        "LoadBalancerPort": 443,
                        "InstanceProtocol": "HTTPS",
                        "InstancePort": 31390,
                        "SSLCertificateId": "arn:aws:acm:us-east-1:709693308983:certificate/<REDACTED>"
                    },
                    "PolicyNames": [
                        "ELBSecurityPolicy-2016-08"
                    ]
                },
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 31400,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 31400
                    },
                    "PolicyNames": []
                },
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 15011,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 31337
                    },
                    "PolicyNames": []
                },
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 8060,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 30341
                    },
                    "PolicyNames": []
                },
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 15030,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 30834
                    },
                    "PolicyNames": []
                },
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 15031,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 30600
                    },
                    "PolicyNames": []
                }
            ],
            "Policies": {
                "AppCookieStickinessPolicies": [],
                "LBCookieStickinessPolicies": [],
                "OtherPolicies": [
                    "ELBSecurityPolicy-2016-08"
                ]
            },
            "BackendServerDescriptions": [],
            "AvailabilityZones": [
                "us-east-1a",
                "us-east-1c",
                "us-east-1d"
            ],
            "Subnets": [
                "subnet-975ffdcb",
                "subnet-9a12beb4",
                "subnet-b696e7fc"
            ],
            "VPCId": "vpc-9e0ef9e4",
            "Instances": [
                {
                    "InstanceId": "i-021a786c4362a0d6a"
                },
                {
                    "InstanceId": "i-02b69c0d561dc27bc"
                },
                {
                    "InstanceId": "i-02caf5296ecedad61"
                },
                {
                    "InstanceId": "i-0681d728a799591a6"
                },
                {
                    "InstanceId": "i-0b2102f062df52bc0"
                },
                {
                    "InstanceId": "i-0e0fa4a4d802135cd"
                }
            ],
            "HealthCheck": {
                "Target": "TCP:31390",
                "Interval": 10,
                "Timeout": 5,
                "UnhealthyThreshold": 6,
                "HealthyThreshold": 2
            },
            "SourceSecurityGroup": {
                "OwnerAlias": "709693308983",
                "GroupName": "k8s-elb-a2abbdb149e2a11e8a6950acd49a3f59"
            },
            "SecurityGroups": [
                "sg-06cbef25aae63b86e"
            ],
            "CreatedTime": "2018-08-12T12:21:00.750Z",
            "Scheme": "internet-facing"
        }
    ]
}

This is my gateway configuration:

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*.example.com"
  - port:
      number: 443
      name: http
      protocol: HTTP
    hosts:
    - "*.example.com"
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - "*.example.com"
  gateways:
  - httpbin
  http:
  - match:
    - port: 80
    route:
      - destination:
          host: httpbin.default.svc.cluster.local
          port:
            number: 8000
  - match:
    - port: 443
    route:
      - destination:
          host: httpbin.default.svc.cluster.local
          port:
            number: 8000

Other facts:

  • All the ports are open from the ELB security group to the kubernetes workers
  • HTTPS works at the ELB level without termination ie. if I use 31380 over tcp as a backend
  • All HTTP connections to the application work from the remote

Any help is greatly appreciated. Please let me know if you need additional information!

UPDATE

it seems a problem with sni? The following configuration works all fine

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*.example.com"
  - port:
      number: 443
      name: http
      protocol: HTTP
    hosts:
    - "*"                          // NOTICE only * instead of *.example.com
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
claudio-benfatto picture claudio-benfatto on 13 Aug 2018
👍3

@subnova Could you check if it's help you? https://github.com/istio/istio.github.io/pull/2321#discussion_r211398844
We've configured TLS ingress gateway for multiple hosts, with non unique port name in the gateway.yaml and we've received SSL_ERROR_SYSCALL

jsenon picture jsenon on 20 Aug 2018

@frankbu @jsenon I am having the same issue. 

K8s version: 1.9
ISTIO Version: 1.0.1

K8s cluster have E load balancer enabled.

I followed this tutorial https://istio.io/docs/tasks/traffic-management/secure-ingress/ and my curl was not working if I only have port 443. It was giving me error: SSL_ERROR_SYSCALL

curl -v -HHost:ms-1234.byom-i344382.eu-central-1.x.x.com https://ms-1234.byom-i344382.eu-central-1.x.x.com:443/api/v2/predict --cacert /home/vagrant/exmaple/mtls-go-example/2_intermediate/certs/ca-chain.cert.pem

then I added port 80 and redirected the traffic to https and it started working. 

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ms-is-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    tls:
      httpsRedirect: true
    hosts:
    - "*.byom-i344382.eu-central-1.x.x.com"
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - "*.byom-i344382.eu-central-1.x.x.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ms-is
spec:
  hosts:
  - "*.byom-i344382.eu-central-1.x.x.com"
  gateways:
  - ms-is-gateway
  http:
  - match:
    - uri:
        exact: /api/v2/predict
    route:
    - destination:
        host: mlf-is.default.svc.cluster.local
        port:
          number: 53547
  - match:
    - headers:
        content-type:
          exact: application/grpc
    route:
    - destination:
        host: mlf-is.default.svc.cluster.local
        port:
          number: 9000
srishti0190 picture srishti0190 on 4 Sep 2018
👍2

Also, while it was not working I tested with certmanager enabled, deployed CertificateIssuer, Certificate and certmanager gateway resources. And It was working. But I dont really need it.

srishti0190 picture srishti0190 on 4 Sep 2018

I just had this very same error. I had to Gateways deployed, both with separate hosts and the same wildcard cert.

After deleting the gateway that was created first TLS to the second service started to work.
The previous 2 gateways were in separate namespaces, I am not sure if it matters or not.

I now run a single gateway with a *.domain.com host and routes based on virtual services.

LinAnt picture LinAnt on 19 Sep 2018

I think everybody on this thread is not having the same problem. There is more than one potential issue.

@LinAnt It sounds like maybe you are running into this issue: https://istio.io/help/ops/traffic-management/deploy-guidelines/#configuring-multiple-tls-hosts-in-a-gateway

The issue that @srishti0190 and others mention seems very different. If I'm understanding it, there is only one Gateway involved, and it only seems to work if there is both an HTTP and HTTPS port defined. Is that right? Are there any potentially related messages in the pilot discovery container log?

frankbu picture frankbu on 19 Sep 2018

[istio-1.0.2 (⎈ |cluster-2:default)]$ kubectl get po -n istio-system
NAME READY STATUS RESTARTS AGE
istio-citadel-86c9878bd9-skn8t 1/1 Running 0 46m
istio-egressgateway-6c7949d598-55b2r 1/1 Running 0 46m
istio-galley-656946f9d-vm65g 1/1 Running 0 46m
istio-ingressgateway-699794bf9c-lgb2q 1/1 Running 0 46m
istio-pilot-5f757f67d8-wwql8 2/2 Running 0 46m
istio-policy-6978f6fff4-wjpr8 2/2 Running 0 46m
istio-sidecar-injector-58996b77bd-5q4p7 1/1 Running 0 46m
istio-statsd-prom-bridge-55965ff9c8-8wh6p 1/1 Running 0 46m
istio-telemetry-bb6b55b67-pc79c 2/2 Running 0 46m
prometheus-7456f56c96-4hkmv 1/1 Running 0 46m
[istio-1.0.2 (⎈ |cluster-2:default)]$ kubectl describe svc istio-ingressgateway -n istio-system

Name: istio-ingressgateway
Namespace: istio-system
Labels: app=istio-ingressgateway
chart=gateways-1.0.1
heritage=Tiller
istio=ingressgateway
release=wishing-llama
Annotations:
Selector: app=istio-ingressgateway,istio=ingressgateway
Type: LoadBalancer
IP: 10.63.242.204
LoadBalancer Ingress: 35.233.137.254
Port: http2 80/TCP
TargetPort: 80/TCP
NodePort: http2 31380/TCP
Endpoints: 10.60.1.7:80
Port: https 443/TCP
TargetPort: 443/TCP
NodePort: https 31390/TCP
Endpoints: 10.60.1.7:443
Port: tcp 31400/TCP
TargetPort: 31400/TCP
NodePort: tcp 31400/TCP
Endpoints: 10.60.1.7:31400
Port: tcp-pilot-grpc-tls 15011/TCP
TargetPort: 15011/TCP
NodePort: tcp-pilot-grpc-tls 32603/TCP
Endpoints: 10.60.1.7:15011
Port: tcp-citadel-grpc-tls 8060/TCP
TargetPort: 8060/TCP
NodePort: tcp-citadel-grpc-tls 32050/TCP
Endpoints: 10.60.1.7:8060
Port: tcp-dns-tls 853/TCP
TargetPort: 853/TCP
NodePort: tcp-dns-tls 30567/TCP
Endpoints: 10.60.1.7:853
Port: http2-prometheus 15030/TCP
TargetPort: 15030/TCP
NodePort: http2-prometheus 32011/TCP
Endpoints: 10.60.1.7:15030
Port: http2-grafana 15031/TCP
TargetPort: 15031/TCP
NodePort: http2-grafana 32669/TCP
Endpoints: 10.60.1.7:15031
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 46m service-controller Ensuring load balancer
Normal EnsuredLoadBalancer 45m service-controller Ensured load balancer
[istio-1.0.2 (⎈ |cluster-2:default)]$ curl -v 35.230.58.76

  • Rebuilt URL to: 35.230.58.76/
  • Trying 35.230.58.76...
  • TCP_NODELAY set
  • connect to 35.230.58.76 port 80 failed: Connection refused
  • Failed to connect to 35.230.58.76 port 80: Connection refused
  • Closing connection 0
    curl: (7) Failed to connect to 35.230.58.76 port 80: Connection refused
drobit picture drobit on 15 Oct 2018

This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

stale[bot] picture stale[bot] on 13 Jan 2019

This is still genuinely a problem, and only the wildcard solution suggested by https://github.com/istio/istio/issues/6071#issuecomment-412453757 helps when you want to setup up a mutual TLS ingress gateway by following this guide: https://istio.io/docs/tasks/traffic-management/secure-ingress/

The guide suggests that you can put in a FQDN in the hosts key when you're setting up a https protocol server in the Gateway resource but I've only ever had success in putting a wildcard *.

Note: I've had these problems against istio 1.0.2

themanifold picture themanifold on 1 Feb 2019

Same problem here on 1.0.5. I've tried many different configurations and the only one that works is when both http and https are enabled on the gateway but I don't want anything to be accessible on http. httpRedirect won't suffice as that then causes a "too many redirects" issue when enabling the cloudflare proxy and we absolutely need cloudflare.

edalford11 picture edalford11 on 18 Feb 2019
👍2

I'm on 1.0.6 and encountered the exactly same issue. There're two workarounds I currently found that make tls work:

  1. like @eigokor said, specify httpRedirect and visit http port instead
  2. like @themanifold said, specify hosts key as wildcard * and then tls works without redirection
edussx picture edussx on 7 Mar 2019
👍1

Still same issue on 1.1.0 and same workarounds working describe by @edussx.
Tested with File Mount-Based approach and using Secret Discovery Service.

anthonyhaussman picture anthonyhaussman on 21 Mar 2019
👍2

I'm on 1.0.6 and encountered the exactly same issue. There're two workarounds I currently found that make tls work:

  1. like @eigokor said, specify httpRedirect and visit http port instead
  2. like @themanifold said, specify hosts key as wildcard * and then tls works without redirection

Does anyone found the fix for it?

  1. httpsRedirect=true doesn't works for me, now I cant access the HTTP nor the HTTPS protocol of my application
  2. I can't use "*" for hosts

Please can someone refer me the proper fix

Istio version: 1.1.6
k8s version: 1.13.6

venomwaqar picture venomwaqar on 30 May 2019

@venomwaqar I have other walk arounds for this. Just add --resolve flag if you're using curl:

--resolve HOST:PORT:ADDRESS  Force resolve of HOST:PORT to ADDRESS

Configuring a dns record for that IP should also work.

I guess it's an SNI-related issue so either of:

  1. specifying hostname using --resolve
  2. using dns
  3. http redirect
  4. have a wildcard cert and specify hosts as *

makes gateway know which cert it should select. Therefore I think it's not an Istio-related bug.

edussx picture edussx on 30 May 2019

I am seeing the same issue where the httpsRedirect: true option is not working at all and HTTP traffic is not being redirected to HTTPS.

Does anyone have any idea what might be causing this and how to fix it?

azalenski picture azalenski on 21 Jun 2019

I still have the same issue in version 1.2.2.
When I try to set a specific host in my Gateway, I have an SSL_ERROR_SYSCALL but it works great with a wildcard.
I have a simple application who returns a token on a POST.
Here are my tests.
Set up working with hosts wildcard:

### With wildcard host
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: test-gateway
  namespace: test
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 443
      name: https-test
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: "wildcard-test-cmp-gg" ## Wildcard certificate *.test.cmp.gg
    hosts:
    - "*"
---


[~]> curl --resolve 'graphql-istio.test.cmp.gg:31390:kube-01.dev.cmp.com'  -vvv --header 'Host: graphql-istio.test.cmp.gg' -kX POST -d 'grant_type=password&[email protected]&password=ThePassord&client_id=1234&client_secret=1234' https://kube-01.dev.cmp.com:31390/oauth/token
Note: Unnecessary use of -X or --request, POST is already inferred.
* Resolve address 'kube-01.dev.cmp.com' found illegal!
* Couldn't parse CURLOPT_RESOLVE entry 'graphql-istio.test.cmp.gg:31390:kube-01.dev.cmp.com'!
*   Trying 192.168.1.1:31390...
* TCP_NODELAY set
* Connected to kube-01.dev.cmp.com (192.168.1.1) port 31390 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.test.cmp.gg
*  start date: May  2 21:16:09 2019 GMT
*  expire date: Jul 31 21:16:09 2019 GMT
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55f2123fbf30)
> POST /oauth/token HTTP/2
> Host: graphql-istio.test.cmp.gg
> User-Agent: curl/7.65.1
> Accept: */*
> content-type: application/x-www-form-urlencoded
> Content-Length: 221
> 
* We are completely uploaded and fine
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
< HTTP/2 200 
< server: istio-envoy
< date: Fri, 05 Jul 2019 08:36:46 GMT
< content-type: application/json; charset=utf-8
< content-length: 1200
< cache-control: no-store
< pragma: no-cache
< x-envoy-upstream-service-time: 230
< backend-response-time: 227
< name: oauth
< version: 1
< endpoint: /oauth/token
< x-datadog-sampling-priority: -1
< 
{"access_token": OK}%

Set up not working with specific hosts:

### With specific host
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: test-gateway
  namespace: test
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 443
      name: https-test
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: "wildcard-test-cmp-gg" ## Wildcard certificate *.test.cmp.gg
    hosts:
    - "graphql-istio.test.cmp.gg"
---

[~]> curl --resolve 'graphql-istio.test.cmp.gg:31390:192.168.1.1' -vvv --header 'Host: graphql-istio.test.cmp.gg' -kX POST -d 'grant_type=password&[email protected]&password=ThePassord&client_id=1234&client_secret=1234' https://kube-01.dev.cmp.com:31390/oauth/token
Note: Unnecessary use of -X or --request, POST is already inferred.
* Added graphql-istio.test.cmp.gg:31390:192.168.1.1 to DNS cache
*   Trying 192.168.1.1:31390...
* TCP_NODELAY set
* Connected to kube-01.dev.cmp.com (192.168.1.1) port 31390 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to kube-01.dev.cmp.com:31390 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to kube-01.dev.cmp.com:31390

I am not interested to use the httpsRedirect method (I don't have http endpoint).
Do we have a status about this problem? SNI issue?

anthonyhaussman picture anthonyhaussman on 8 Jul 2019
👍1

@anthonyhaussman Please note that the host in the Gateway must match the host in the Virtual Service. So when you change the host name in your Gateway from wildcard to a specific host, please remember to change it in the VirtualService as well.

vadimeisenbergibm picture vadimeisenbergibm on 8 Jul 2019

@vadimeisenbergibm It was the case.
But shame on me, the problem was coming from my curl command.

The wrong command:
[~]> curl --resolve 'graphql-istio.test.cmp.gg:31390:192.168.1.1' -vvv --header 'Host: graphql-istio.test.cmp.gg' -kX POST -d 'grant_type=password&[email protected]&password=ThePassord&client_id=1234&client_secret=1234'      https://kube-01.dev.cmp.com:31390/oauth/token

The right command:
[~]> curl --resolve 'graphql-istio.test.cmp.gg:31390:192.168.1.1' -vvv --header 'Host: graphql-istio.test.cmp.gg' -kX POST -d 'grant_type=password&[email protected]&password=ThePassord&client_id=1234&client_secret=1234'      https://graphql-istio.test.cmp.gg:31390/oauth/token

So it working great. :ok_hand:

anthonyhaussman picture anthonyhaussman on 16 Jul 2019

@anthonyhaussman I am glad it worked for you :)

vadimeisenbergibm picture vadimeisenbergibm on 16 Jul 2019
👍1

Seems the issue is fixed. If there are still problems please reopen -- its a big bug so maybe I missed something

howardjohn picture howardjohn on 30 Aug 2019

I don't think this is fixed yet, or at least I cannot confirm that is is fixed. I will try the other bits of advice in this post but I'm not aware of any code changes that have been made to fix my problem: https://github.com/istio/istio/issues/6071#issuecomment-459745577

themanifold picture themanifold on 3 Sep 2019

Up for this ^ still having the same issue also

spacejam101 picture spacejam101 on 29 Oct 2019

@themanifold @spacejam101 this issue contains multiple problems reported by multiple people, some of the problems seem to be resolved. Could you please open new issues and describe the problems you experience? Then we will need to check your configuration, ask you to dump pilot configuration, and so on, so it will be the question of your specific configuration, your environment and your pilot configuration and logs.

vadimeisenbergibm picture vadimeisenbergibm on 29 Oct 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Stono picture Stono  Â·  92Comments
lanceliuu picture lanceliuu  Â·  82Comments
Stono picture Stono  Â·  65Comments
ldemailly picture ldemailly  Â·  87Comments
hzxuzhonghu picture hzxuzhonghu  Â·  96Comments