Aws-app-mesh-roadmap: Bug: Impossible to connect to external https service

[piontas]

Summary
We have a service running in kubernetes (EKS). Service is also defined in App Mesh as virtual Node and Virtual Service. Service is able to communicate with other services in App Mesh (http or tcp connections). We have external services (APIs) that we want to consume. Obviously all APIs are running over HTTPS. External services are defined as Virtual Nodes and Virtual Services in App Mesh (According to docs). Whenever service is trying to connect to External https service, envoy is not allowing this traffic with (sometimes random) SSL errors. All aws services over https are working fine: eks api, x-ray, kms, s3, etc.

Steps to Reproduce

1. Create App Mesh with DROP_ALL egress filter

1. Define Virtual Node for external service

{
    "meshName": "test-mesh",
    "spec": {
        "listeners": [
            {
                "portMapping": {
                    "port": 443,
                    "protocol": "tcp"
                }
            }
        ],
        "serviceDiscovery": {
            "dns": {
                "serviceName": "google.com"
            }
        }
    },
    "virtualNodeName": "google-test"
}

1. Define Virtual Service

{
    "meshName": "test-mesh",
    "spec": {
        "provider": {
            "virtualNode": {
                "virtualNodeName": "google-test"
            }
        }
    },
    "virtualServiceName": "google.com"
}

1. Define virtual service as backend for test service

{
    "meshName": "test-mesh",
    "spec": {
        "backends": [
            "google.com"
        ],
        "listeners": [
            {
                "portMapping": {
                    "port": 8080,
                    "protocol": "http"
                }
            }
        ],
        "serviceDiscovery": {
            "dns": {
                "serviceName": "service.local"
            }
        }
    },
    "virtualNodeName": "test-service"
}

1. Try to reach https://google.com from your service

when trying to use curl:
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to google.com:443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to google.com:443

Are you currently working around this issue?
APPMESH_EGRESS_IGNORED_PORTS: "443" - of course it's not a good/secure and production ready solution.

Additional context
We have tried with and with and without routers, with defining only virtual node without virtual service even. We have tried different listeners: tcp, https, https.

Envoy logs:

envoy logs:
[2019-11-26 16:15:52.939][1][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:72] starting async DNS resolution for google.com
[2019-11-26 16:15:52.944][1][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:79] async DNS resolution complete for google.com
[2019-11-26 16:15:53.065][19][debug][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:18] original_dst: New connection accepted
[2019-11-26 16:15:53.065][19][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:72] tls inspector: new connection accepted
[2019-11-26 16:15:53.353][19][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:142] tls:onServerName(), requestedServerName: google.com
[2019-11-26 16:15:53.354][19][debug][conn_handler] [source/server/connection_handler_impl.cc:313] closing connection: no matching filter chain found
[2019-11-26 16:15:53.569][1][debug][main] [source/server/server.cc:175] flushing stats

Example errors from java service:

• When using http or http2:

javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

• When using tcp:

java.net.SocketException: Connection reset

Related issues:

• https://github.com/aws/aws-app-mesh-examples/issues/62
• https://github.com/aws/aws-app-mesh-roadmap/issues/2

[kiranmeduri]

I tried to repro with www.google.com [FAIL], www.amazon.com [SUCCESS], www.nike.com [SUCCESS] using the config similar to what you provided. However google.com, amazon.com, nike.com all fail (possibly) due to 301 redirect, so there is certainly some issue to dive into.

1. external-service virtual-node with listener (port=443, protocol=tcp)
2. external-service virtual-service with provider as virtual-node
3. app virtual-node with backend as external-service

For me www.google.com fails due to IPv6 resolution of DNS (#121).

aws appmesh describe-virtual-node --mesh howto-k8s-external-service --virtual-node-name nike-howto-k8s-external-service
{
    "virtualNode": {
        "meshName": "howto-k8s-external-service",
        "metadata": {
            ...
        },
        "spec": {
            "backends": [],
            "listeners": [
                {
                    "portMapping": {
                        "port": 443,
                        "protocol": "tcp"
                    }
                }
            ],
            "serviceDiscovery": {
                "dns": {
                    "hostname": "www.nike.com"
                }
            }
        },
        "status": {
            "status": "ACTIVE"
        },
        "virtualNodeName": "nike-howto-k8s-external-service"
    }
}


aws appmesh describe-virtual-service --mesh howto-k8s-external-service --virtual-service-name www.nike.com
{
    "virtualService": {
        "meshName": "howto-k8s-external-service",
        "metadata": {
            ...
        },
        "spec": {
            "provider": {
                "virtualRouter": {
                    "virtualRouterName": "nike-howto-k8s-external-service"
                }
            }
        },
        "status": {
            "status": "ACTIVE"
        },
        "virtualServiceName": "www.nike.com"
    }
}

aws appmesh describe-virtual-node --mesh howto-k8s-external-service --virtual-node-name front-howto-k8s-external-service
{
    "virtualNode": {
        "meshName": "howto-k8s-external-service",
        "metadata": {                                                                                                                                 
             ...
        },
        "spec": {
            "backends": [
                {
                    "virtualService": {
                        "virtualServiceName": "www.nike.com"
                    }
                }
            ],
            "listeners": [
                {
                    "portMapping": {
                        "port": 8080,
                        "protocol": "http"
                    }
                }
            ],
            "serviceDiscovery": {
                "dns": {
                    "hostname": "front.howto-k8s-external-service.svc.cluster.local"
                }
            }
        },
        "status": {
            "status": "ACTIVE"
        },
        "virtualNodeName": "front-howto-k8s-external-service"
    }
}

Thanks

[piontas]

I have used google.com as an example here. It’s failing also when we try to reach our API’s running behind ALB (IPv4) in different AWS account in the same region. Everything works fine when 443 is ignored on App Mesh.

[kiranmeduri]

@piontas I am not seeing the same error. Can you provide envoy config_dump (listeners) if possible?

[bcelenza]

So from my perspective, even if the connectivity could be made, we aren't doing enough to provide the validation of the TLS session to the external service.

For example, if you choose to model this as TCP, your application is responsible for negotiating TLS with the external service, but that doesn't really meet our goal of providing the service mesh abstraction where the application is no longer responsible.

For modelling the external service as HTTP, this proposal should provide the necessary inputs for negotiating TLS to the service, which would resolve the specific error you see when modelling google as HTTP.

[piontas]

I still don't see it solves a problem, I know that my application should handle TLS negotiation and it does. It works properly when 443 is ignored on App Mesh. I will provide more logs and config_dump @kiranmeduri, as we discussed at our last meeting.

[bcelenza]

I still don't see it solves a problem, I know that my application should handle TLS negotiation and it does. It works properly when 443 is ignored on App Mesh. I will provide more logs and config_dump @kiranmeduri, as we discussed at our last meeting.

For my own understanding: Are you saying that you want the application (not the proxy) to perform TLS negotiation?

To provide a little more context on modelling via HTTP with client policies, which would allow the proxy to negotiate TLS on the application's behalf, here's an example of how it would work:

1. You would model the external service as a virtual node just as you did above, but use http as the protocol instead:

{
    "meshName": "test-mesh",
    "spec": {
        "listeners": [
            {
                "portMapping": {
                    "port": 443,
                    "protocol": "http"
                }
            }
        ],
        "serviceDiscovery": {
            "dns": {
                "serviceName": "google.com"
            }
        }
    },
    "virtualNodeName": "google-test"
}

On the virtual node you call that external service from, you would model a client policy that is configured to negotiate TLS:

{
    "meshName": "test-mesh",
    "spec": {
        "backends": [
            {"virtualService": {
                 "virtualServiceName": "google.com",
                 "clientPolicy": {
                     "tls": {
                         "validation": {
                             "trust": {
                                 // (note: your default trust bundle location may be different)
                                 "file": { "certificateChain": "/etc/pki/tls/cert.pem" } 
                             }
                         }
                     }
                 }
            }}
        ],
        "listeners": [
            {
                "portMapping": {
                    "port": 8080,
                    "protocol": "http"
                }
            }
        ],
        "serviceDiscovery": {
            "dns": {
                "serviceName": "service.local"
            }
        }
    },
    "virtualNodeName": "test-service"
}

Hope that provides some clarity.

[bcelenza]

We've heard from enough customers that modeling external destinations in App Mesh is a bit too tedious, and doesn't provide the right level of support for use cases such as agents/other side cars that require TLS negotiation. I'm closing this issue in favor of #2, where TLS support will be considered as well, and hope to have a public proposal on that issue in the next week or two.

[patricknazar]

I've tried working around the issue by ignoring port 443 but it doesn't work for me. I've spend two days trying to do this and nothing anywhere works, so frustrating.

[patricknazar]

When I use EgressIgnoredPorts = 443 I get "Cannot assign requested address" with curl. Why?! All I want to do is call a https url from my service, I've spend days nothing works. Is there anyone out there that can help?!

[cgchinmay]

@patricknazar Hi, Could you upload your spec equivalent and explain little more on what you are trying to do ? Like your mesh config and url you are trying to access. Thanks

[patricknazar]

@patricknazar Hi, Could you upload your spec and explain little more on what you are trying to do ? Like your mesh config and url you are trying to access. Thanks

Thanks for your reply, I'm about to lose the plot.

It's so very simple. I have a Mesh with Egress filter: Allow external traffic. In there I have a virtual service with a virtual node running on ECS. The service needs to access a https URL, but it gets disconnected by the envoy proxy. All HTTPS connections fail. I've ssh'd into the ECS instance and executed curl -v https://google.com inside the service docker container:

* Expire in 0 ms for 6 (transfer 0x5628cf97af90)
* Expire in 1 ms for 1 (transfer 0x5628cf97af90)
* Expire in 0 ms for 1 (transfer 0x5628cf97af90)
* Expire in 2 ms for 1 (transfer 0x5628cf97af90)
* Expire in 0 ms for 1 (transfer 0x5628cf97af90)
* Expire in 0 ms for 1 (transfer 0x5628cf97af90)
* Expire in 2 ms for 1 (transfer 0x5628cf97af90)
* Expire in 0 ms for 1 (transfer 0x5628cf97af90)
* Expire in 0 ms for 1 (transfer 0x5628cf97af90)
* Expire in 2 ms for 1 (transfer 0x5628cf97af90)
* Expire in 1 ms for 1 (transfer 0x5628cf97af90)
* Expire in 1 ms for 1 (transfer 0x5628cf97af90)
* Expire in 4 ms for 1 (transfer 0x5628cf97af90)
* Expire in 1 ms for 1 (transfer 0x5628cf97af90)
* Expire in 2 ms for 1 (transfer 0x5628cf97af90)
* Expire in 2 ms for 1 (transfer 0x5628cf97af90)
*   Trying 216.58.196.142...
* TCP_NODELAY set
* Expire in 149996 ms for 3 (transfer 0x5628cf97af90)
* Expire in 200 ms for 4 (transfer 0x5628cf97af90)
* Connected to google.com (216.58.196.142) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to google.com:443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to google.com:443

same thing for other https urls. If I try with http:

* Expire in 0 ms for 6 (transfer 0x5563f4fc8f90)
* Expire in 1 ms for 1 (transfer 0x5563f4fc8f90)
* Expire in 0 ms for 1 (transfer 0x5563f4fc8f90)
* Expire in 1 ms for 1 (transfer 0x5563f4fc8f90)
* Expire in 0 ms for 1 (transfer 0x5563f4fc8f90)
* Expire in 0 ms for 1 (transfer 0x5563f4fc8f90)
* Expire in 2 ms for 1 (transfer 0x5563f4fc8f90)
* Expire in 0 ms for 1 (transfer 0x5563f4fc8f90)
* Expire in 0 ms for 1 (transfer 0x5563f4fc8f90)
* Expire in 2 ms for 1 (transfer 0x5563f4fc8f90)
* Expire in 1 ms for 1 (transfer 0x5563f4fc8f90)
* Expire in 1 ms for 1 (transfer 0x5563f4fc8f90)
* Expire in 4 ms for 1 (transfer 0x5563f4fc8f90)
* Expire in 1 ms for 1 (transfer 0x5563f4fc8f90)
* Expire in 1 ms for 1 (transfer 0x5563f4fc8f90)
* Expire in 2 ms for 1 (transfer 0x5563f4fc8f90)
*   Trying 216.58.203.110...
* TCP_NODELAY set
* Expire in 149996 ms for 3 (transfer 0x5563f4fc8f90)
* Expire in 200 ms for 4 (transfer 0x5563f4fc8f90)
* Connected to google.com (216.58.203.110) port 80 (#0)
> GET / HTTP/1.1
> Host: google.com
> User-Agent: curl/7.64.0
> Accept: */*
>
>
...several seconds of delay...
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

I've tried modelling the external host as a virtual service+node and a tcp/443 backend but no difference. I've tried using backend http/443 but I get ssl3_get_record:wrong version number but this isn't going to be a solution anyway

Sigh :'(

[cgchinmay]

Based on what you are trying to do, it look like you are facing TLS termination Issue. You need to add a TLS client policy to your virtual node and let envoy handle TLS for you. However not all https URL's work, if there is a redirection (301) then the envoy is not capable of handling it yet and there is an ongoing work item: https://github.com/aws/aws-app-mesh-roadmap/issues/207
I don't know if google.com is the exact url you are trying to curl but here is a working example for github.com and you can try this approach for your https url.

From your application container : curl -v http://github.com:443/ (You need to use http else you would see the above SSL errors)

---
apiVersion: v1
kind: Namespace
metadata:
  name: my-ns
  labels:
    mesh: apps-mesh
    appmesh.k8s.aws/sidecarInjectorWebhook: enabled
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: Mesh
metadata:
  name: apps-mesh
spec:
  namespaceSelector:
    matchLabels:
      mesh: apps-mesh
  egressFilter: 
    type: ALLOW_ALL
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: front-node
  namespace: my-ns
spec:
  podSelector:
    matchLabels:
      app: front
  listeners:
    - portMapping:
        port: 80
        protocol: http
  serviceDiscovery:
    dns:
      hostname: front.my-ns.svc.cluster.local
  backends:
    - virtualService:
        virtualServiceRef:
          name: github
        clientPolicy:
           tls:
               enforce: true
               validation:
                 trust:
                   file:
                     certificateChain: "/etc/ssl/certs/ca-bundle.crt"    
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: github-node
  namespace: my-ns
spec:
  listeners:
    - portMapping:
        port: 443
        protocol: http
  serviceDiscovery:
    dns:
      hostname: 'github.com'
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualService
metadata:
  name: github
  namespace: my-ns
spec:
  awsName: github.com
  provider:
    virtualNode:
      virtualNodeRef:
        name: github-node

For multiple https backends, this will be the change in the front virtual node

backends:
    - virtualService:
        virtualServiceRef:
          name: github
    - virtualService:
        virtualServiceRef:
          name: <some other service using https as well>
  backendDefaults:
    clientPolicy:
           tls:
               enforce: true
               validation:
                 trust:
                   file:
                     certificateChain: "/etc/ssl/certs/ca-bundle.crt"

[patricknazar]

@cgchinmay I appreciate the response.

I did try with the client policy as written in an earlier comment (with /etc/pki/tls/cert.pem) however there was a problem with that, which I was going to mention in my last comment but didn't: I cannot change the URL in question (to make https=http and add :443 at the end) because the URL is determined by a 3rd-party library. In fact, it makes the request entirely. To be more specific its a PHP auth0 library which is requesting a URL like https://xxxx.auth0.com/.well-known/jwks.json. What option do I have now?

Also, does this explain why the non-https curl command times out?

[patricknazar]

@cgchinmay I tried it manually anyway curl -v http://xxxx.auth0.com:443/.well-known/jwks.json and it times out:

> GET /.well-known/jwks.json HTTP/1.1
> Host: xx.auth0.com:443
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 504 Gateway Timeout
< content-length: 24
< content-type: text/plain
< date: Wed, 24 Feb 2021 07:12:30 GMT
< server: envoy
<
* Connection #0 to host xx.auth0.com left intact
upstream request timeout

[cgchinmay]

I will take a look at envoy config_dump, configured clusters and envoy logs for the above URL to determine what could be the problem and get back to you.Meanwhile could you upload your yaml spec equivalent, to understand how you are configuring the virtual node and virtual service to access above URL ?

[patricknazar]

I will take a look at envoy config_dump, configured clusters and envoy logs for the above URL to determine what could be the problem and get back to you.Meanwhile could you upload your yaml spec, to understand how you are configuring the virtual node and virtual service to access above URL ?

That is not the real URL I replaced my auth0 tenant with xx in the URL. But honestly, the same problem occurs for https://google.com. We can use that for now. I'm using terraform for this particular infrastructure and it's split across several files and modules, and used with different variables. Not to mention some values I've changed manually as I'm experimenting with the correct solution. So it's quite difficult to copy paste here, I'll try to include and translate the relevant parts:

For the ecs task definition:

proxy_configuration {
    type           = "APPMESH"
    container_name = var.envoy_container_name
    properties = {
      AppPorts           = var.app_ports # "80"
      EgressIgnoredIPs   = "169.254.170.2,169.254.169.254"
      IgnoredUID         = "1337"
      ProxyEgressPort    = 15001
      ProxyIngressPort   = 15000
    }
  }

For the modelled auth0 service

resource "aws_appmesh_virtual_service" "auth0" {
  name       = var.service_name # "auth0"
  mesh_name  = var.mesh_id
  mesh_owner = var.mesh_owner

  spec {
    provider {
      virtual_node {
        virtual_node_name = aws_appmesh_virtual_node.auth0.name
      }
    }
  }
}

For the modelled auth0 node

resource "aws_appmesh_virtual_node" "auth0" {
  name       = var.node_name # "auth0"
  mesh_name  = var.mesh_id
  mesh_owner = var.mesh_owner

  spec {
    listener {
      port_mapping {
        port     = var.service_port #443
        protocol = var.node_protocol #tcp
      }
    }



    service_discovery {
      dns {
        hostname = var.hostname # "xxx.auth0.com"
      }
    }
  }
}

For the service node (its a http api):

resource "aws_appmesh_virtual_node" "this" {
  name       = var.node_name
  mesh_name  = var.mesh_id
  mesh_owner = var.mesh_owner

  spec {
    listener {
      port_mapping {
        port     = var.service_port #80
        protocol = var.node_protocol #http
      }
    }

    dynamic "backend" {
      for_each = var.backends # ["auth0"]

      content {
        virtual_service {
          virtual_service_name = backend.value
        }
      }
    }

    backend_defaults {
      client_policy {}
    }

    logging {
      access_log {
        file {
          path = "/dev/stdout"
        }
      }
    }

    service_discovery {
      aws_cloud_map {
        service_name   = var.cloudmap_service_name
        namespace_name = var.cloudmap_namespace_name
      }
    }
  }
}

Hopefully this helps. I really appreciate you trying to help me with this, thanks.

[cgchinmay]

Without exact URL, it's hard for me to replicate. Cannot use google.com since it could have other problems. Might not be related to be the same as yours. So will need to check envoy logs. This is what you can do
1.Port forward traffic from envoy to localhost:9901

1. Increase envoy verbosity level with this curl request

curl -X POST "http://localhost:9901/logging?level=debug"

3.Check logs for the envoy in other terminal as you make the curl request

Envoy logs should tell you what's happening with the URL or any connection issues. Upload this information. Also check Config_dump and Clusters information found at localhost:9901. See how the endpoints for your https. url are configured. Upload this information.

[patricknazar]

@cgchinmay You can try with https://patricknazar-test.au.auth0.com/.well-known/jwks.json

I'll see if I can try that, do you know how to port forward using ECS? I don't use k8s

[patricknazar]

Ok I've tried that but literally no logs came up as I was doing the request.

[efe-selcuk]

@patricknazar Sorry you're having issues here. Accessing external services is not where we want it to be, we have a couple of outstanding feature requests like #2 around this.

In general, please create a new issue if you need help with a problem. Commenting on a closed issue is far less visible.

Application-originated TLS is the most restrictive situation, because it can't be terminated at the local proxy, so at best Envoy can route the raw TCP traffic based on the port; if you have multiple backends listening on 443, it can't route at all. That's why we suggest http backend and an explicit client policy. Though #207 can become a problem.

It's curious that ALLOW_ALL egress filter isn't working either. I happened to come across this today while investigating a different IPv6-related issue. https://farcaller.net/jwks-remote-fetch-is-failed/

121 may be affecting you, and/or your VPC and ECS tasks aren't configured with IPv6 support (assuming you're running into the same issue as that article, which should show up in debug logs).

[patricknazar]

@efe-selcuk Thanks for that! I didn't create a new issue because I saw this one with basically the same symptoms.

I've checked out my trace logs and I get some similarity to https://farcaller.net/jwks-remote-fetch-is-failed/ but with an immediate connection error: 99 code instead of 101:

[2021-02-25 01:21:32.507][19][debug][connection] [source/common/network/connection_impl.cc:769] [C1248] connecting to [2606:4700::6812:ec2d]:443
[2021-02-25 01:21:32.507][19][debug][connection] [source/common/network/connection_impl.cc:789] [C1248] immediate connection error: 99
[2021-02-25 01:21:32.507][19][debug][pool] [source/common/tcp/original_conn_pool.cc:125] queueing request due to no available connections
[2021-02-25 01:21:32.508][19][debug][conn_handler] [source/server/connection_handler_impl.cc:476] [C1247] new connection
[2021-02-25 01:21:32.508][19][trace][connection] [source/common/network/connection_impl.cc:508] [C1247] socket event: 2
[2021-02-25 01:21:32.508][19][trace][connection] [source/common/network/connection_impl.cc:616] [C1247] write ready
[2021-02-25 01:21:32.508][19][trace][connection] [source/common/network/connection_impl.cc:508] [C1248] socket event: 3
[2021-02-25 01:21:32.509][19][debug][connection] [source/common/network/connection_impl.cc:519] [C1248] raising immediate error
[2021-02-25 01:21:32.509][19][debug][connection] [source/common/network/connection_impl.cc:203] [C1248] closing socket: 0
[2021-02-25 01:21:32.509][19][debug][pool] [source/common/tcp/original_conn_pool.cc:140] [C1248] client disconnected

Let's say I want to try this recommended fix of dns_lookup_family: V4_ONLY how can I do that with app mesh? Is that even possible?

I seem to be getting a lot of

[2021-02-25 01:48:18.117][1][trace][connection] [source/common/network/connection_impl.cc:546] [C1566] read ready. dispatch_buffered_data=true
[2021-02-25 01:48:18.117][1][trace][connection] [source/common/network/raw_buffer_socket.cc:39] [C1566] read error: Resource temporarily unavailable

[efe-selcuk]

121 is what would cover that ipv4/ipv6 dns, so at the moment there isn't a workaround for that specific issue. If you're savvy and want to experiment, you could set up an envoy proxy in a local docker container just to toy with sending traffic to your auth0 endpoint, using the configuration that app mesh is pushing to it. But of course, that's not going to reflect reality.

I suspect there's a combination of the IPv6 DNS issue, in addition to ECS and VPC settings that are preventing the traffic at all.

[patricknazar]

@efe-selcuk It begs the question, how is anyone, anywhere using ECS, app mesh and calling external services successfully. Seems odd. Surely they have just enough difference to my setup that they don't run into the issue. If there's something I can do to work around it I'd be keen.

[efe-selcuk]

I don't have much data on how prevalent external service modeling is with app mesh customers, and things become trickier if you don't own that external endpoint. For the few cases that I've directly assisted with, either modeling the tcp/443 backend (when there's only one) or http/443 has gotten folks unblocked. But the redirect and IPv6 issues have blocked others.

If the endpoint publishes AAAA records and Envoy is preferring those IPv6 records, then it's potentially solvable by creating dual-stack ECS tasks, and configuring your VPC to be IPv6-friendly to let the proxy talk to that endpoint. This is untested.

[efe-selcuk]

You can also use the ALLOW_ALL egress filter on the mesh, that's the one that skipped my mind. It's a bit of a heavy hand, but it'll allow you to originate your TLS traffic from your application, resolve ipv4 (probably), and let envoy just pass it through. That unblocks some use-cases.

[patricknazar]

You can also use the ALLOW_ALL egress filter on the mesh, that's the one that skipped my mind. It's a bit of a heavy hand, but it'll allow you to originate your TLS traffic from your application, resolve ipv4 (probably), and let envoy just pass it through. That unblocks some use-cases.

I am using ALLOW_ALL already :(

[efe-selcuk]

Ah, right sorry, wires crossed. Presumably you're able to curl the endpoint from your tasks if envoy is not present?

Make sure you don't have a backend present that matches that name while you're trying to use ALLOW_ALL. Right now we don't intercept application-originated IPv6 traffic at all, so if it's getting to Envoy, your app resolved an IPv4 address. If the egress filter is enabled, and you don't have a backend that matches the name, envoy would let the traffic through to its original destination.

If you're on Slack, you can reach me on our public slack and I can take a look at your envoy's config dump.

[patricknazar]

I fixed my problem by

• enabling egress ALLOW_ALL of course
• disabling ipv6 on the ECS instance with user data sysctl commands so that the tasks won't use any IPv6 DNS records (this was a problem with auth0)
• using a NAT instance. I'm sure NAT gateway is easier/better, but too expensive for us atm. The instance setup had its own peculiarities with iptables that I won't go into

Thanks to all who helped

[sathish-kumar-narayanan]

Does any got this working with egressFilter: type: DROP_ALL

I am trying to connect to the azure storage service from a pod, and upload a file. It works fine in non mesh namespace,
Can someone point me on what am doing wrong ?

curl -v --upload-file logger.txt http://<name1>.blob.core.windows.net:443/assets/75252601-67c1-4707-bf7e-c89f41a9544a\?comp\=block\&blockid\=ZGQ3ZTU4M2ItMDAwMDAx\&ss\=b\&sp\=wc\&sv\=2018-03-28\&sig\=<sig>\&srt\=o\&se\=2021-11-10T12%3A58Z\&api-version\=2018-03-28

*   Trying xx.xxx.152.74...
* TCP_NODELAY set
* Connected to <name1>.blob.core.windows.net (xx.xxx.152.74) port 443 (#0)
> PUT /assets/75252601-67c1-4707-bf7e-c89f41a9544a?comp=block&blockid=<id>&ss=b&sp=wc&sv=2018-03-28&sig=<sig>&srt=o&se=2021-11-10T12%3A58Z&api-version=2018-03-28 HTTP/1.1
> Host: <name1>.blob.core.windows.net:443
> User-Agent: curl/7.52.1
> Accept: */*
> Content-Length: 2821
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 404 Not Found
< date: Thu, 18 Mar 2021 12:32:57 GMT
< server: envoy
< connection: close
< content-length: 0
< 
* we are done reading and this is set to close, stop send
* Curl_http_done: called premature == 0
* Closing connection 0

apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
  labels:
    app: ecoax
    um-service: um-asset-upload
    version: 2-7-0-004-dae0241
  name: um-asset-upload-2-7-0-004-dae0241
  namespace: um-studio
spec:
  backendDefaults:
    clientPolicy:
      tls:
        enforce: true
        validation:
          trust:
            file:
              certificateChain: /etc/pki/tls/cert.pem
  backends:
  - virtualService:
      virtualServiceRef:
        name: azure-storage-service-1
  - virtualService:
      virtualServiceRef:
        name: azure-storage-service-2
  listeners:
  - portMapping:
      port: 8080
      protocol: http
  podSelector:
    matchLabels:
      app: ecoax
      um-service: um-asset-upload
      version: 2-7-0-004-dae0241
  serviceDiscovery:
    dns:
      hostname: um-asset-upload-2-7-0-004-dae0241.um-studio.svc.cluster.local
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualService
metadata:
  name: azure-storage-service-1
spec:
  awsName: azure-storage-service-1.um-studio.svc.cluster.local
  provider:
    virtualNode:
      virtualNodeRef:
        name: azure-storage-node-1
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualService
metadata:
  name: azure-storage-service-2
spec:
  awsName: azure-storage-service-2.um-studio.svc.cluster.local
  provider:
    virtualNode:
      virtualNodeRef:
        name: azure-storage-node-2
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: azure-storage-node-1
spec:
  listeners:
  - portMapping:
      port: 443
      protocol: http
  serviceDiscovery:
    dns:
      hostname: <name1>.blob.core.windows.net
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: azure-storage-node-2
spec:
  listeners:
  - portMapping:
      port: 443
      protocol: http
  serviceDiscovery:
    dns:
      hostname: <name2>.blob.core.windows.net
---

[saleem-mirza]

I am requesting to re-open this defect rather than closing it. I'm facing same issue where appmesh is resolving IPv6 address and throwing error upstream connect error or disconnect/reset before headers. reset reason: connection failureroot@<pod_name>

please vote

[rajal-amzn]

@saleem-mirza AppMesh doesnt support IPV6 address yet. Please see additional details here: #314 and #121.

[sathish-kumar-narayanan]

@rajal-amzn In my case, I tried connecting to the same HTTPS backend with mesh egress ALLOW_ALL, and without any backend definitions in virtual service, and It works fine. But when I add the backend def, and tried to access the service via http://<>:443, it is not working.

Could you help me with this ?