Aws-app-mesh-roadmap: Bring Envoy from official release

[jamsajones]

You should be able to use the offical release of Envoy.

[NoOrdInaryGuy]

@jamsajones Out of interest, do you have any info about what modifications are in aws-appmesh-envoy:v1.8.0.2-beta versus the official Envoy release? (If this is something you can disclose).

[lavignes]

@NoOrdInaryGuy
We're tracking the progress of up-streaming the main modifications in #38 and #39:

• #38 An xDS authentication extension for using IAM credentials to connect to our envoy management service.
• #39 An X-Ray tracer extension.

[NoOrdInaryGuy]

Great, that's good to know - thanks for the info!

[CharlyF]

👋 all, just out of curiosity - Would you have an ETA on this one ?
If I remember correctly, I was able to use the Datadog tracer from envoy a few months back using your custom image (I assume it backported this change https://github.com/envoyproxy/envoy/pull/4699 among others).
Trying to make it work again, I am seeing:

[2019-03-14 01:04:43.372][1][info][config] source/server/configuration_impl.cc:95] loading tracing configuration
[2019-03-14 01:04:43.372][1][info][config] source/server/configuration_impl.cc:104]   loading tracing driver: envoy.tracers.datadog
[2019-03-14 01:04:43.372][1][critical][main] source/server/server.cc:80] error initializing configuration '/tmp/envoy.yaml': Didn't find a registered implementation for name: 'envoy.tracers.datadog'
[2019-03-14 01:04:43.372][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:89] Client teardown, resetting streams

Which leads me to believe that the implementation is not available in this current image (v1.8.0.2-beta).
I think this along with some fixes was merged after 1.8.0 and included in the official 1.9.0, do you know if a new aws-envoy image will be build (prior to GA) with this feature (backported or just bumped to 1.9.0) ?
Best,
.C

[shubharao]

@CharlyF We now support version v1.9.1.0. We do not yet have an ETA for upstreaming the changes we made (primarily for Sigv4 authentication of Envoy), but this is on our high priority item list

[tsykora-verimatrix]

Is the v1.9.1.0 available outside us-west-2?

"Failed to enable App Mesh integration
Envoy container must use the official App Mesh Envoy image"

when I try to use:
111345817488.dkr.ecr.us-east-1.amazonaws.com/aws-appmesh-envoy:v1.9.1.0-prod

[lavignes]

Hi @tsykora-verimatrix

We recommend that you use the latest Envoy image available, which is currently tagged as v1.11.1.1-prod: https://docs.aws.amazon.com/app-mesh/latest/userguide/envoy.html

It is available in all the App Mesh regions. Do you need to use v1.9.1.0 in particular?

[kamaldon]

I am attempting to you use the version v1.11.1.1 specified in the documentation but I am getting the same error
Image: 111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-envoy:v1.11.1.1-prod
"Failed to enable App Mesh integration
Envoy container must use the official App Mesh Envoy image"

[tsykora-verimatrix]

111345817488.dkr.ecr.us-east-1.amazonaws.com/aws-appmesh-envoy:v1.11.1.1-prod

Status reason | CannotPullContainerError: Error response from daemon: pull access denied for 111345817488.dkr.ecr.us-east-1.amazonaws.com/aws-appmesh-envoy, repository does not exist or may require 'docker login'
-- | --
1337

@lavignes doesn't work either

[bcelenza]

@tsykora-verimatrix Just to confirm:

1. Your ECS task is running in us-east-1 as well?
2. You have given the ECS task permissions to pull images from ECR? (see these docs)

I ran a quick test to ensure that the ECR repository was open for pulling from all accounts:

$(aws ecr --region us-east-1 get-login --no-include-email --registry-ids 111345817488)

docker pull 111345817488.dkr.ecr.us-east-1.amazonaws.com/aws-appmesh-envoy:v1.11.1.1-prod

v1.11.1.1-prod: Pulling from aws-appmesh-envoy
72d97abdfae3: Already exists 
083db2dc1aa4: Pull complete 
0ea0eef1b868: Pull complete 
97858b908079: Pull complete 
cb999a5ecfd0: Pull complete 
753534a07b32: Pull complete 
Digest: sha256:afea680c7ea35fd886be0c8599beb8d9285c2d62732480cd19a9ecda65291c13
Status: Downloaded newer image for 111345817488.dkr.ecr.us-east-1.amazonaws.com/aws-appmesh-envoy:v1.11.1.1-prod

To be very sure your account can pull, you could try to run the commands above.

[bcelenza]

@kamaldon I was able to reproduce your issue on the ECS console, which is where I assume you are seeing it?

Update: I'm not actually able to reproduce this. I had personally forgotten to click the Apply button on the App Mesh integration. Is it possible you've run into the same thing?

[kamaldon]

@bcelenza My original attempt was trying to configure it through the "Configure via JSON" option which caused the failure. I tried inputting version 1.11.1 into the "Envoy Image" text field and I also clicked the Apply button and there are no errors, and I can even create the new revision but it doesn't actually update the docker image. The new task definition revision still has the 1.9.0 version.

For some reason the Apply button keeps defaulting it back to 1.9.0.

[tsykora-verimatrix]

@tsykora-verimatrix Just to confirm:

1. Your ECS task is running in us-east-1 as well?

2. You have given the ECS task permissions to pull images from ECR? (see [these docs](https://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_on_ECS.html))

I ran a quick test to ensure that the ECR repository was open for pulling from all accounts:

$(aws ecr --region us-east-1 get-login --no-include-email --registry-ids 111345817488)

docker pull 111345817488.dkr.ecr.us-east-1.amazonaws.com/aws-appmesh-envoy:v1.11.1.1-prod

v1.11.1.1-prod: Pulling from aws-appmesh-envoy
72d97abdfae3: Already exists 
083db2dc1aa4: Pull complete 
0ea0eef1b868: Pull complete 
97858b908079: Pull complete 
cb999a5ecfd0: Pull complete 
753534a07b32: Pull complete 
Digest: sha256:afea680c7ea35fd886be0c8599beb8d9285c2d62732480cd19a9ecda65291c13
Status: Downloaded newer image for 111345817488.dkr.ecr.us-east-1.amazonaws.com/aws-appmesh-envoy:v1.11.1.1-prod

To be very sure your account can pull, you could try to run the commands above.

thank you, indeed my IamTaskExecRole (automatically generated by CDK) was limited to my service container only, after extending it for envoy container the Fargate service I created was able to pull the envoy container.

So Envoy is now running as side-car container with my application.

Last missing piece of the puzzle is the missing Proxy/ initContainer functionality without which I can't really pass any traffic to AppMesh service. (I'm not using AWS console, instead I use CDK).

[bcelenza]

@kamaldon Yup, I'm able to reproduce that issue (Apply still sets as 1.9.0). Working on getting this resolved. Thanks for the clarification!

[bcelenza]

@tsykora-verimatrix Looks like you've already found it, but support for proxyConfiguration in the CDK is logged as https://github.com/aws/aws-cdk/issues/3977.

[jamsajones]

We are done with the work required to allow App Mesh to work with Envoy upstream. Officially App Mesh auth via SigV4 will be available when milestone 1.12.0 completes (see: https://github.com/envoyproxy/envoy/issues/8042). Outstanding work is X-Ray related and I am closing this issue in favor of #21 and #95.