Aws-app-mesh-roadmap: Use App Mesh for ingress routing

[jamsajones]

[zbintliff]

Finding this issue saved me some time. Was deep diving documentation to see if we could use it for ingress. Currently planning on using Envoy for that as well, but missing the control plane piece.

[coopr]

@zbintliff if you didn't find it in the spot in the docs you'd expect to see it, feel free to PR an improvement to the docs 😄 - TIA!

[zbintliff]

Definitely. Does that mean my understanding from above correct?

On Fri, Nov 30, 2018, 1:31 PM Cooper Marcus [e-mail]
wrote:

@zbintliff https://github.com/zbintliff if you didn't find it in the
spot in the docs you'd expect to see it, feel free to PR an improvement to
the docs 😄 - TIA!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/awslabs/aws-app-mesh-examples/issues/31#issuecomment-443296123,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHLWl4LczGOD0S9bGPeG1KOyrHJsXer2ks5u0Xl7gaJpZM4Y4M2b
.

[coopr]

My understanding is that your understanding is correct @zbintliff - AWS App Mesh does not (yet) support ingress routing or typical "edge" API gateway functions. I'm happy to be corrected if I'm mistaken 😉

[jamsajones]

@zbintliff and @coopr you are correct.

Service to service communication has been our focus at this point. We do not do anything to keep customers from deploying an Envoy that would work as an ingress router, but we have not tested it ourselves, do not have any good examples for it and so there could be subtle gotchas. Your continued input here will help us understand and prioritize this feature. If you feel that you have any interesting use cases around this feature please let us know. Cheers!

[pawelprazak]

any plans for API Gateway integration?

[FractalJim]

Lack of ingress routing is the main thing holding us back from using AppMesh at the moment. Without ingress routing we need to maintain separately managed config for ingress routing as well as taking on AppMesh config, making the benefits over doing your own envoy deployment on EKS less clear. With ingress routing AppMesh is a clear winner.

[rverma-nikiai]

@jamsajones Does it meant that we can't make alb ingress communicate with virtual services/nodes.
Also is it possible to expose a service publically if that is deployed as part of app-mesh?

[mumoshu]

In case this "App Mesh for Ingress Routing" works like a managed Envoy cluster controllable via App Mesh API, I'd definitely love it.

I'll use it as an alternative to ALB, because it gives us gRPC load-balancing/routing which isn't possible with ALB.

[manikawnth-abg]

@jamsajones Does it meant that we can't make alb ingress communicate with virtual services/nodes.
Also is it possible to expose a service publically if that is deployed as part of app-mesh?

I think as long as you attach ALB while creating your ecs service, the target group can act as service registry for ALB ingress. This can be done along with your service discovery (rt 53) while creating ecs service, which is used for within mesh communication.
Checkout the colorteller demo of appmesh
Folks can correct me if I'm wrong.

[rverma-nikiai]

@manikawnth-abg I am not very clear on this. Anyways we are trying to run app-mesh on eks. Currently, we have target groups with health check behind ALB load balancer.
When I injected app-mesh sidecar to those containers the health check started failing. It's simply that now the load balancers are not able to connect to the pod by IP.

Can you please expand your answer in this direction?

[kiranmeduri]

@rverma-nikiai that seems like a misconfiguration or a bug. Can you please open a new issue where we can dive deep into it. Thanks

[rverma-nikiai]

@kiranmeduri just to ensure what I am doing is correct, I created an ingress (say hello.aws.com) using aws-alb-ingress-controller. Then created a virtual service(hello) and a virtual node(hello.aws.com with service discovery as DNS) and a virtual route along with standard kubernetes service & pod. Added egress filter as allow all for now.

Doing the above steps I observed two things.

1. ALB can't do health checks
2. ALB traffic is not coming to pod.

There is no concept of gateway unlike istio and I also couldn't find an example in eks which demonstrate how to expose a service over https publically.
Appreciate your help.

Also wondering if we compulsorily require cloudmap to work with app mesh, since all our services are within eks cluster and only outside resources are rds, dynamo, sqs, redis?

[kiranmeduri]

@rverma-nikiai When Envoy is injected into the pod, it intercepts all the incoming and outgoing traffic from the application container. This should be true for health-check too.

• Can you confirm that health-check using curl against a candidate pod is also failing?

• Next can you see that Envoy has correct cluster definition (distributed by appmesh via xDS)

kubectl exec -it $POD_ID -c envoy curl http://localhost:9901/clusters | cut -f1 -d":" | sort | uniq

Above command should show proper clusters configured.

• If you see clusters then can you share envoy access logs on how the health-checks are failing.

kubectl logs -f $POD_ID -c envoy 

Thanks

[tomaszdudek7]

+1, I need this badly.

As referenced in #119, this feature would enable to do canary deployments internally + externally at the same time. Right now ALB does not mix in nicely and we have to run our own Virtual Node(nginx, HAproxy, zuul or self-made ColorGateway in AWS ColorApp example) on the edge to do so.

That is both time consuming(configuring nginx or zuul, creating our own gateway), dangerous(needs to have HA etc.) and prone to human errors(misconfiguration nginx vs Virtual Routers).

I would love(and pay) AWS to take off this burden from my team

[rverma-nikiai]

I believe the best place for edge proxy is route53 and with its support for weighted routing. IWith Alb-ingress-controller(latest release) we can have two app-groups v1, v2. All the edge services mapped to virtual node as usual. The missing component is the external-dns support for weighted routing https://github.com/kubernetes-incubator/external-dns/issues/196, but the PR branch work.

I am guessing this would be more performant as well. If anyone can foresee any issues with this approach would love to hear.

With this approach, there are two main advantages for us, WAF and Cognito :)

[mumoshu]

@rverma-nikiai The solution you proposed works, as long as you use HTTP only.

For HTTP/2 and gRPC(#13), DNS-level weighting doesn't work as they use persistent connections that can be much longer than the TTL of your DNS records.

Of course I'm not saying your solution doesn't work. It is just that we still need something that doesn't depend on DNS/Route 53 for h2 and grpc :)

[kiranmeduri]

Referencing API Gateway proposal #111

[tomaszdudek7]

@shubharao, could you please tell us what does it mean "Phase: Researching"? Are you guys actually working on it? If yes, do you plan to support ALB?

I see the proposal about API Gateway, but unfortunately not all of us utilize it. I'm certain many ALB users would like to use App Mesh as ingress to the entire stack.

[rajal-amzn]

Yes we are actively working on enabling ingress for App mesh. It will be natively supported by App Mesh where it can be either an ECS or EKS service.

[tomaszdudek7]

Does that Include ALB + ECS combo? It looks like ALB + EKS is already possible with Gloo(https://aws.amazon.com/blogs/containers/using-gloo-as-an-ingress-gateway-for-aws-app-mesh/) for example.

[rajal-amzn]

Hey all, Here is our proposal for enabling Ingress for App Mesh.
We propose to define 2 new AppMesh resources - Virtual Gateway and Gateway Route

Gateway Route

Gateway Routes are similar to Routes in the sense that they also have configurations to specify HTTP request match rules but they target a backend Virtual Service instead of a Virtual Node in the case of Routes. Also, the Gateway Routes are not associated to Virtual Routers.

Virtual Gateway

Virtual Gateways will act as Ingress to the App Mesh and would allow a set of services to be accessed from outside the mesh under a common domain. Virtual Gateways would be associated to an ECS/Fargate/EC2 service or an EKS Pod running a set of standalone Envoys. Virtual Gateways will listen on a port from which it receives traffic from the internet, terminate the TLS using a defined certificate. Then it would inspect the HTTP request headers, use the rules defined in the Gateway Route to decide the target Virtual Service to forward the request. Before forwarding the request to backend Virtual Service, it can initiate a TLS request based on the configurations.

Example

If the Virtual Gateway is configured to expose for "www.example.com" and if the Gateway Route is configured to accept /books requests from Virtual Gateway to Virtual Service A (provider could be Virtual Node or Virtual Router), on receiving a request for "www.example.com/books", the request would be forwarded to Virtual Service A.
Here is an example call to create a HTTP Gateway Route:

aws appmesh create-gateway-route --mesh-name sample-mesh \
    --gateway-route-name ingress-books-route \
    --spec
{
            "httpRoute" : {
                    "headers" : [ 
                    {
                        "invert" : false, 
                        "match" : 
                        {
                            "exact" : "exact", 
                            "prefix" : "prefix", 
                            "range" :
                            {
                                "end" : 10, 
                                "start" : 5
                            }, 
                            "regex" : "regex", 
                            "suffix" : "suffix" 
                        }, 
                        "name" : "name"
                    }
                    ], 
                "match" : {
                    "method" : "GET", 
                    "prefix" : "/books"
                },
                "action" : {
                    "target" : {
                        "virtualService": {
                            "virtualServiceName" : "books-virtual-service",
                        }
                    }
                }
           }
}

Here is an example call to create a HTTP2 Gateway Route:

aws appmesh create-gateway-route --mesh-name sample-mesh \
    --gateway-route-name ingress-books-route \
    --spec
{
            "http2Route" : {
                    "headers" : [ 
                    {
                        "invert" : false, 
                        "match" : 
                        {
                            "exact" : "exact", 
                            "prefix" : "prefix", 
                            "range" :
                            {
                                "end" : 10, 
                                "start" : 5
                            }, 
                            "regex" : "regex", 
                            "suffix" : "suffix" 
                        }, 
                        "name" : "name"
                    }
                    ], 
                "match" : {
                    "method" : "GET", 
                    "prefix" : "/books"
                },
                "action" : {
                    "target" : {
                        "virtualService": {
                            "virtualServiceName" : "books-virtual-service",
                        }
                    }
                }
           }
}

Here is an example call to create a GRPC Gateway Route:

aws appmesh create-gateway-route --mesh-name sample-mesh \
    --gateway-route-name ingress-books-route \
    --spec
{
            "grpcRoute" : {
                 "match" : {
                    "metadata" : [
                    {
                        "invert" : false,
                        "name" : "name",
                        "match" : {
                            "exact" : "exact", 
                            "prefix" : "prefix", 
                            "range" :
                            {
                                "end" : 10, 
                                "start" : 5
                            }, 
                            "regex" : "regex", 
                            "suffix" : "suffix" 
                        }
                    }
                    ],
                    "methodName": "method",
                    "serviceName" : "service"
                },
                "action" : {
                    "target" : {
                        "virtualService": {
                            "virtualServiceName" : "books-virtual-service",
                        }
                    }
                }
           }
}

Here is an example call to create a Virtual Gateway:

aws appmesh create-virtual-gateway --mesh-name sample-mesh \
    --virtual-gateway-name ingress-gateway \
    --spec
{
            "backends" : [ 
                {
                    "gatewayRoute" : 
                    {
                        "priority": 0,
                        "gatewayRouteName" : "ingress-books-route",
                        "clientPolicy": {
                            "tls" : {
                                //To initiate TLS to this particular route. Will override the backendDefaults
                                "enforce": true,
                                "ports": [443],
                                "validation": {
                                    "trust": {
                                        "acm": {
                                            "certificateAuthorityArns": [
                                                "arn:aws:acm:us-west-2:abc:certificate-authority/id"
                                            ]
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            ],
            "backendDefaults" : {
                "clientPolicy" : {
                    "tls" : {
                        //Default configuration for initiating TLS to all backends
                       "enforce": false,
                    }
                }
            }, 
            "listeners" : [  
            { 
                "healthCheck" :
                {
                   "healthyThreshold" : 10, 
                   "intervalMillis" : 5, 
                   "path" : "path/to/health_check_request", 
                   "port" : 800, 
                   "protocol" : "http", 
                   "timeoutMillis" : 10, 
                   "unhealthyThreshold" : 3     
                },
                "portMapping" :
                {
                    "port" : 8080, 
                    "protocol" : "http"  
                },
                "logging" : 
                {
                    "accessLog" : {
                        "file" : {
                            "path" : "/path/to/access.log"
                        }
                    }
                },
                "tls": 
                {
                    //For terminating TLS request
                    "mode": "STRICT",
                    "certificates": [
                    {
                        "sds": {
                            "secretName": "foo-certificate",
                            "source": {
                                "unixDomainSocket": {
                                    "path": "/path/to/sds-server.sock"
                                }
                            }
                        } 
                    }
                    ]
                } 
            }
            ]
}

NOTE: The TLS configuration in backend client policy, backendDefaults client policy for initiating TLS and TLS termination defined in TLS will follow the same configurations as defined in here and here

How does the Gateway Route work with Routes defined for a Virtual Service?

We now have Gateway routes from VirtualGateway to Virtual Service which can further have routes to Virtual nodes through a Virtual Router. In that case, the 2 routes will be appended. For example:
If we assume the following resources:

aws appmesh create-virtual-router --mesh-name sample-mesh \
    --virtual-router-name books-router \
    --spec
{
    "listeners": [
    {
        "portMapping": 
        {
            "port": 8010,
            "protocol": "HTTP"
        }
    }
    ]
}

aws appmesh create-route --mesh-name sample-mesh \
    --route-name reviews-route --virtual-router-name books-router \
    --spec
{
                "match" : {
                    "method" : "GET", 
                    "prefix" : "/reviews"
                },
                "action" : {
                    "target" : {
                        "weightedTargets": [{
                            "virtualNode" : "book-reviews-node",
                            "weight": 100
                        }]
                    }
                }
}

aws appmesh create-virtual-service --mesh-name sample-mesh \
    --virtual-service-name books-virtual-service \
    --spec
{
    "provider": 
    {
        "virtualRouter": 
        {
            "virtualRouterName": "books-router"
        }
    }
}
aws appmesh create-gateway-route --mesh-name sample-mesh \
    --gateway-route-name ingress-books-route \
    --spec
{
            "httpRoute" : {
                "match" : {
                    "method" : "GET", 
                    "prefix" : "/books"
                },
                "action" : {
                    "target" : {
                        "virtualService": {
                            "virtualServiceName" : "books-virtual-service",
                        }
                    }
                }
           }
}
aws appmesh create-virtual-gateway --mesh-name sample-mesh \
    --virtual-gateway-name ingress-gateway \
    --spec
{
            "backends" : [ 
                {
                    "gatewayRoute" : 
                    {
                        "priority": 0,
                        "gatewayRouteName" : "ingress-books-route",
                    }
                }
            ],
            "listeners" : [  
            { 
                "portMapping" :
                {
                    "port" : 8080, 
                    "protocol" : "http"  
                }
            }
            ]
}

In this case, Virtual Gateway would forward “www.example.com/books/reviews” to Virtual Node "book-reviews-node"

Summary

We hope these changes fit your use cases for ingress. We'll be updating this issue once we have the feature enabled in our Preview Channel. Until then, we'd love to hear your feedback on this proposal in the comments.

Thanks!

[benbpyle]

Could it work as another option for a Private Link like a NLB does behind API Gateway? Or would you still have an NLB as the private link and just use it as a pass-through to the ingress traffic in App Mesh?

[rverma-nikiai]

Does virtual gateway will use loadbalancers like ALB/NLB. Will it support cognito?
From the example it seems ssl termination using public ACM is not supported currently, is it?

[manikawnth]

@rajal-amzn

Virtual Gateways would be associated to an ECS/Fargate/EC2 service or an EKS Pod running a set of standalone Envoys. Virtual Gateways will listen on a port from which it receives traffic from the internet, terminate the TLS using a defined certificate.

Now this becoming more like Istio and we're loosing cool abstraction that could've been provided by aws. Since Gateway registers to what is called an "edge proxy" of envoy, I would also suggest the ability (optionally) to create a ingress gateway (ECS service or EKS ingress service) only the fly

1. Ability to create an ingress gateway (edge proxy) from appmesh cli (including SSL termination, passthrough, MTLS)
2. Ability to associate to an existing "edge proxy"
3. Ability to associate with an existing ALB / NLB targetgroup. (SSL termination for ALB, termination and passthrough for NLB)
4. Ability to associate with API gateway

If all the above options can be done via appmesh console, that would be cool

[mumoshu]

@manikawnth Just curious, but what's the real benefit of integrating with ALB/NLB/API Gateway here? I believe we'd better start from real issue(s).

For example, my use-case for this feature is to expose my gRPC service running on EKS to other microservices in the same VPC.

I don't particularly love being forced to use NLB or ALB to expose a gateway composed of envoy instances. NLB preserves source IPs so you need to tweak the backend's security group to accept traffic from the client accessing NLB. ALB doesn't support HTTP/2 backends which makes it impossible to front HTTP/2 or gRPC servers load-balanced by the gateway. What I need is really an easy-to-configure and scalable L7 load-balancer that speaks HTTP/2.

For API Gateway, does it even support pod/task IPs(Private IPs assigned to ENIs attached to K8s pods/ECS tasks) as load-balancing targets?

Does we AWS users really understand which ELB offering best match our diverging use-cases? If not, an AppMesh API to allow choosing which ELB offering to integrate really help?

It would be true that once ALB added H/2 backend support having a out-of-box integration with AppMesh + ALB helps. I'm waiting it to happen for a few years but it's still the thing.

I use ELB(classic) or NLB combined with non-appmesh envoy as an ingress gateway to my grpc services today. If we aren't gonna wait ALB to become as feature-rich as appmesh/envoy, I'd rather love having a fully-managed envoy cluster that speaks AppMesh API and integrates nicely with ACM, Route 53, etc, so I can completely replace ALB for my specific use-case.

Under the hood, AWS would certainly leverage existing AWS services like NLB to make it scalable, but I don't care about that as a user.

That's my goal, but I'm not sure that's what's yours and that's why I suggested to start from the real issue(s).

[SleeperSmith]

expose my gRPC service running on EKS to other microservices in the same VPC.

Woudn't you route that in the mesh to begin with?

But for exposing traffic to the public internet, yeah agree with the point on NLB / sec group and ALB/Http2 issues.

[rajal-amzn]

@rverma-nikiai

Does virtual gateway will use loadbalancers like ALB/NLB. Will it support cognito?
From the example it seems ssl termination using public ACM is not supported currently, is it?

Virtual Gateways do not use ALB/NLB by default. You can attach it to ALB/NLB based on your needs.
The first release of Ingress support will not include Cognito support and public certs though we will be working on them in the future.

[rajal-amzn]

@manikawnth @benbpyle
Thanks for your feedback. We are working on the design of integration of Virtual Gateways with API Gateway. We will update this thread as we finalize the details.

[raulcabello]

I'm also interested in this! any update on when it will be available?
Thanks!

[samrui]

@rverma-nikiai

@manikawnth-abg I am not very clear on this. Anyways we are trying to run app-mesh on eks. Currently, we have target groups with health check behind ALB load balancer.
When I injected app-mesh sidecar to those containers the health check started failing. It's simply that now the load balancers are not able to connect to the pod by IP.

Can you please expand your answer in this direction?

I'm facing the same problem. When I add envoy in pod, the health check failed. Do you solve the problem ?

[rajal-amzn]

@samrui

I'm facing the same problem. When I add envoy in pod, the health check failed. Do you solve the problem ?

Can you open up a new issue along with your configuration details?

[samrui]

hi, @rajal-amzn
I have submitted a new issue. #174

[SBerda]

Any update ? :)

[rajal-amzn]

Ingress Gateways are available for testing in our preview channel. Check out the walkthrough to get started, and the docs for more info. We are enforcing authorization of the Envoy Proxy of Virtual Gateways when connecting to App Mesh's Envoy Management Service.

Kindly leave your feedback here.

[eddgrant]

Great to see this hit preview, thanks @rajal-amzn . Would it be possible, either now or in the future, to extend the preview channel to other regions? A current client of mine is generally restricted to in eu-west-2 and would love to test this, but its availability only in us-west-2 makes this rather more difficult. Being able to test this in eu-west-2 would reduce the number of hoops we have to jump through. :pray:

[rajal-amzn]

@eddgrant Thanks for your feedback. I see that building preview for a different region is not in our roadmap. Added it now

[eddgrant]

Thanks @rajal-amzn , much appreciated :+1:

[rajal-amzn]

Hey all, here is the API proposal for K8s integration. We have a design proposal for v2 CRD which defines the AppMesh resources in a kubernetes-native way. Since that is in active development, we are defining Ingress resources based on the v2 scheme.

User scenarios

• As a Cluster Administrator, I should be able to create/maintain the Virtual Gateway for the mesh.
• As a Cluster Administrator, I should be able to whitelist one/more namespaces to create Gateway routes for the Virtual Gateway.
• As an Application developer, I should be able to define and maintain the routes that reach my application from the Gateway.

Design

We propose to add 2 new CRDs:

• VirtualGateway: Represents an AppMesh VirtualGateway object
• GatewayRoute: Represents an AppMesh GatewayRoute object

Virtual Gateway

Virtual Gateway is namespace-scoped and will be associated to the mesh based on the namespaceSelector field defined in the mesh. In the below example, any Virtual Gateway defined in namespace my-gateway-ns would get associated to the mesh my-mesh

apiVersion: appmesh.k8s.aws/v1beta2
kind: Mesh
metadata:
  name: my-mesh
spec:
  awsName: my-cluster-mesh
  namespaceSelector:
    matchLabels:
      mesh: my-mesh
----
apiVersion: v1
kind: Namespace
metadata:
  name: my-gateway-ns
  labels:
    mesh: my-mesh
----

• Virtual Gateway resource has awsName field to denote the AppMesh name for the Virtual Gateway. It defaults to ${name}_${namespace} when not specified.
• Virtual Gateway resource has a namespaceSelector field that defines which Gateway Routes are associated to it. In future, we could add fine-grained selectors.
  NOTE: The label selectors of two Virtual Gateways must not overlap. Controller should error if detected conflicting Virtual Gateways for given Gateway Route resource.

apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualGateway
metadata:
  name: my-gateway
  namespace: my-gateway-ns
spec:
  awsName: my-gateway_my-gateway-ns
  namespaceSelector:
    matchLabels:
      gateway: my-gateway
  - listeners:
    - portMapping:
        port: 9080
        protocol: http
      logging:
        accessLog:
          file:
            path: /dev/stdout
      ## For terminating TLS request
      tls:
        mode: STRICT
        certificate:
          file:
            certificateChain: "/path/to/cert-chain.pem"
            privateKey: "/path/to/private-key.pem"
  backendDefaults:
    clientPolicy:
      ## For initiating TLS when communicating with backend services
      tls:
        enforce: true
        ports: [443]
        validation:
          trust:
            acm:
              certificateAuthorityArns:
              - "arn:aws:acm:us-west-2:abc:certificate-authority/id"

  status:
    virtualGatewayArn: "arn:aws:appmesh:us-west-2:virtualGateway/id"

Gateway Route

Gateway Routes are namespace-scoped and we typically expect that the Gateway Routes are defined in the same namespace as the Application. As mentioned above, they would be associated to the Virtual Gateway based on the namespaceSelector field defined in the Virtual Gateway. For the below example, any Gateway Route defined in the namespace my-app-ns would get associated to the Virtual Gateway my-gateway

apiVersion: v1
kind: Namespace
metadata:
  name: my-app-ns
  labels:
    gateway: my-virtual-gateway
    mesh: my-mesh

• Gateway Route resource has awsName field to denote the AppMesh name for it. It defaults to ${name}_${namespace} when not specified.
• Gateway Route resource specifies the target Virtual Service using the virtualServiceRef field. The namespace field is optional if the target Virtual Service is in the same namespace. The below example shows declaration of a sample HTTP/HTTP2/GRPC Gateway Route.

##Http Gateway Route
apiVersion: appmesh.k8s.aws/v1beta2
kind: GatewayRoute
metadata:
 name: http-gateway-route
 namespace: my-app-ns
spec:
  awsName: http-gateway-route_my-app-ns
  http:
    match:
      prefix: "/httpService"
    action:
      target:
          virtualServiceRef:
            namespace: my-app-ns
            name: httpServiceName
  status:
    gatewayRouteArn: "arn:aws:appmesh:us-west-2:gatewayRoute/id1"
---
##Http2 Gateway Route
apiVersion: appmesh.k8s.aws/v1beta2
kind: GatewayRoute
metadata:
  name: http2-gateway-route
  namespace: my-app-ns
spec:
  awsName: http2-gateway-route_my-app-ns
  http2:
    match:
      prefix: "/http2Service"
    action:
      target:
          virtualServiceRef:
            namespace: my-app-ns
            name: http2Service
  status:
    gatewayRouteArn: "arn:aws:appmesh:us-west-2:gatewayRoute/id2"

---
##Grpc Gateway Route
apiVersion: appmesh.k8s.aws/v1beta2
kind: GatewayRoute
metadata:
  name: grpc-gateway-route
  namespace: my-app-ns
spec:
  awsName: grpc-gateway-route_my-app-ns
  grpc:
    match:
      serviceName: "grpcService"
    action:
      target:
          virtualServiceRef:
            namespace: my-app-ns
            name: grpcServiceName
  status:
    gatewayRouteArn: "arn:aws:appmesh:us-west-2:gatewayRoute/id3"

Please leave your feedback on the comments if you have any.

[eddgrant]

Hi folks,

If I've understood the approach correctly, each time a service is deployed in to the mesh it is necessary to add a new route to the VirtualGateway (assuming we want the new service to be accessible from outside of the mesh)?

If this is the case, it would be it would be really helpful if route management on the virtual gateways was idempotent. This would avoid us having to know whether we should be creating or updating a route each time a service is deployed. We could instead be completely stateless and just "attach" the route each time. Would be great if you could give some guidance as to whether this is something that has been considered yet?

Also, I appreciate this is still in preview, but was wondering if you're able to give any hints as to whether route management will end up getting CloudFormation support? We currently manage all our deployments in CloudFormation and it would be great to be able to attach routes within our CF stacks.

Cheers,

Edd

[efe-selcuk]

@eddgrant Thanks for your comment.

each time a service is deployed in to the mesh it is necessary to add a new route to the VirtualGateway

If I follow the rest of the message, deploy here refers to a net-new Virtual Service. In that case, a new route (or a modification of a route) would naturally be needed, or an existing route would need to be modified to point to the new VS. After that, unless you're changing routing, the gateway route shouldn't need to be modified for new revisions of that service.

it would be really helpful if route management on the virtual gateways was idempotent

The idea of an "attach" primitive is interesting. We've focused on providing declarative CRUD primitives and having Virtual Services serve as the "smart pointer". What is your use-case such that you'd need to update a gateway route on each service revision? Given that you're already using CloudFormation, what is missing, given that it will already decide on create vs update for you? Just looking for more information on what you would want your integration to look like.

Re: CloudFormation, we will indeed support Virtual Gateways and Gateway Routes via CloudFormation. They'll typically arrive shortly after the feature is fully released.

[eddgrant]

Hi @efe-selcuk ,

Thanks for taking the time to respond and apologies for taking so long to respond myself.

Great to hear that CloudFormation support will be coming. My reference to the "attach" primitive was really my thinking about how the various create and update API calls might be orchestrated, but thinking about it now, that's exactly what CloudFormation does already, so I think what you're proposing would actually work for us.

We're looking forward to the release of this in CloudFormation, cheers!

[herbertgoto]

AWS CLI 2.0 Support

Summary
Using AWS CLI 2.0 you cannot add the model for the preview channel.

Steps to Reproduce
Using AWS CLI 2.0.0, I execute the following command:

aws configure add-model \
    --service-name appmesh-preview \
    --service-model https://raw.githubusercontent.com/aws/aws-app-mesh-roadmap/master/appmesh-preview/service-model.json

It returns the following error:

Expecting value: line 1 column 1 (char 0)

Are you currently working around this issue?
Yes. By using AWS CLI 1.17.7

[herbertgoto]

I follow instructions in: https://docs.aws.amazon.com/app-mesh/latest/userguide/virtual_gateways.html

When I try to create the virtual-gateway I get an error. I have tried with:

aws appmesh-preview create-virtual-gateway --cli-input-json file://virtual-gateway.json

aws appmesh-preview create-virtual-gateway --mesh-name appmesh-service-discovery --virtual-gateway-name myVGW --spec file://virtual-gateway.json

The error I get is:

An error occurred (NotFoundException) when calling the CreateVirtualGateway operation: Mesh with name vgw is not found under account

If I call aws appmesh list-meshes, I can see the meshName I am using.

[efe-selcuk]

@herbertgoto Is it the first or second command that fails? In the second, you're passing in appmesh-service-discovery as the mesh name, which of course is not the mesh vgw in your error message.

Also, could you please report that CLI 2.0 problem as a separate issue?

[rajal-amzn]

@herbertgoto We have an open issue to track the support for AWS CLI 2.0 support.

When you create AppMesh resources using aws appmesh-preview, the resources are created against our preview environment whereas the resources created using aws appmesh are created against our prod environment. I guess, you are trying to access the prod resources from preview. If yes, it would not work. Please create a mesh using aws appmesh-preview create-mesh and then use it to create virtual gateway.

[herbertgoto]

@herbertgoto Is it the first or second command that fails? In the second, you're passing in appmesh-service-discovery as the mesh name, which of course is not the mesh vgw in your error message.

Also, could you please report that CLI 2.0 problem as a separate issue?

Both comments have been resolved by @rajal-amzn

[isaac-mj]

Hi guys - I have an upcoming project that could be a good candidate to use virtual gateways. However, I need this feature in production. Do you have an estimation about it? It would help me decide whether we should wait to include it or not.

[rajal-amzn]

@isaac-mj We cannot commit to a date as per policies but I can say that this is our top priority and we are actively working on releasing it in production. You can expect the release soon.

[nightmareze1]

+1 I wait a ingress gw for ECS! Because in kubernetes is easy but in ecs is very complex you.need code you personal api-gateway..

[rajal-amzn]

Hey all,
We will be making a minor change to the VirtualGateway API when we release this feature globally. Initially, we had designed to let customers define _logging_ as part of the _listeners_ object which might help the gateway owners to separate out the logs since Virtual Gateways serve traffic to multiple services. On further analysis from our end, we have decided to pull it out of the listeners and have one common logging per VirtualGateway.
So the updated API model will look as below:

{ 
    "type" : "AWS::AppMesh::VirtualGateway", 
    "properties" : 
    { 
        "meshName" : "SampleMesh", 
        "virtualGatewayName" : "IngressGateway",
        "spec" :
        {
            "backendDefaults" : {
                ...
            }, 
            "listeners" : [  
            { 
                "healthCheck" :
                {
                    ...
                },
                "portMapping" :
                {
                    ...
                },
                "tls":
                {
                    ...
                }
            }
            ],
            "logging" : 
            {
                "accessLog" : {
                    "file" : {
                        "path" : "/path/to/access.log"
                    }
                }
            }
        } 
    } 

We have updated our preview model as per this new API. This would be a breaking change to the models created in preview currently. So, if you have already created the access log, it would continue to work until we release this feature in GA. Post that, you will have to update your Virtual Gateway API to reflect the changes required, in order to get the access log changes working.

[benpettman]

What kind of authentication will this have? OR how can it be used to authenticate requests? Does this have to be done by some other mechanism?

[efe-selcuk]

@benpettman Initially, TLS and x.509 certificates are going to be the authentication mechanism _from_ the gateway _into_ your mesh. We're in research and/or design phase for Mutual TLS (#34) and external authorization (#140), both of which have implications for authentication at the gateway.

Can you tell us more about your use case?

[zaheershaik]

AWS CLI 2.0 Support

Summary
Using AWS CLI 2.0 you cannot add the model for the preview channel.

Steps to Reproduce
Using AWS CLI 2.0.0, I execute the following command:

aws configure add-model \
    --service-name appmesh-preview \
    --service-model https://raw.githubusercontent.com/aws/aws-app-mesh-roadmap/master/appmesh-preview/service-model.json

It returns the following error:

Expecting value: line 1 column 1 (char 0)

Are you currently working around this issue?
Yes. By using AWS CLI 1.17.7

I had the same issue. I was able to work around by downloading the service model json locally and by referring the file using file:// in the add-model command.

[rajal-amzn]

Hey all,
Thank to everyone for your feedback. Ingress Gateways for AppMesh is now generally available. Check out this blog post for more details: https://aws.amazon.com/about-aws/whats-new/2020/07/aws-app-mesh-launches-ingress-support-for-virtual-gateways/

NOTE: Cloudformation support for this is yet to be launched. We will update this issue once it is done.

[rajal-amzn]

Cloudformation support for Virtual Gateway are live. Check the docs for details.

Resolving this issue.