Aws-app-mesh-roadmap: Bug: Underscores are illegal in egress hostnames

[vinhnguyen500]

Summary
Tomcat services are reporting that the egress host name is unparsable because of the underscores.

Steps to Reproduce
Add a web service using Tomcat as its proxy server to App Mesh
Start the service and look at the DEBUG logs.

Are you currently working around this issue?
Doesn't seem to interfere with ingress. Only egress is causing noisy logs.

Additional context
Seems like this is the format that you guys are using for egress host names

• name: cds_egress_${MESH_NAME}_${TARGET_VIRTUAL_NODE_NAME}_http_8080

Could this be changed to hyphens instead?

Attachments

2020-03-11 16:50:31.667 INFO 1 --- [nio-5000-exec-7] o.apache.coyote.http11.Http11Processor : The host [cds_egress_${MESH_NAME}_${TARGET_VIRTUAL_NODE_NAME}_http_5000] is not valid
Note: further occurrences of request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: The character [_] is never valid in a domain name.
at org.apache.tomcat.util.http.parser.HttpParser$DomainParseState.next(HttpParser.java:926) ~[tomcat-embed-core-9.0.21.jar!/:9.0.21]
at org.apache.tomcat.util.http.parser.HttpParser.readHostDomainName(HttpParser.java:822) ~[tomcat-embed-core-9.0.21.jar!/:9.0.21]
at org.apache.tomcat.util.http.parser.Host.parse(Host.java:71) ~[tomcat-embed-core-9.0.21.jar!/:9.0.21]
at org.apache.tomcat.util.http.parser.Host.parse(Host.java:45) ~[tomcat-embed-core-9.0.21.jar!/:9.0.21]
at org.apache.coyote.AbstractProcessor.parseHost(AbstractProcessor.java:288) ~[tomcat-embed-core-9.0.21.jar!/:9.0.21]
at org.apache.coyote.http11.Http11Processor.prepareRequest(Http11Processor.java:810) ~[tomcat-embed-core-9.0.21.jar!/:9.0.21]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:384) ~[tomcat-embed-core-9.0.21.jar!/:9.0.21]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) ~[tomcat-embed-core-9.0.21.jar!/:9.0.21]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:853) ~[tomcat-embed-core-9.0.21.jar!/:9.0.21]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587) ~[tomcat-embed-core-9.0.21.jar!/:9.0.21]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.21.jar!/:9.0.21]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.21.jar!/:9.0.21]
at java.base/java.lang.Thread.run(Thread.java:830) ~[na:na]

[efe-selcuk]

Is there anything you can provide to help us stand up an example to repro this? A cloudformation template and/or an ECS task definition, or something along those lines would be great.

In the meantime I'll still see if I can stand something up.

[vinhnguyen500]

The specific framework I was using was Spring Boot which uses Tomcat underneath the hood. I can't provide the image but the image is using openjdk:latest. I can provide you the pom.xml file though

[vinhnguyen500]

The specific framework I was using was Spring Boot which uses Tomcat underneath the hood. I can't provide the image but the image is using openjdk:latest. I can provide you the pom.xml file though

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
4.0.0

<groupId>com.company</groupId>
<artifactId>service</artifactId>
<version>0.1.0</version>

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.2.1.RELEASE</version>
</parent>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-beans</artifactId>
            <version>5.2.1.RELEASE</version>
    </dependency>
    <dependency>
        <groupId>com.jayway.jsonpath</groupId>
        <artifactId>json-path</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.google.http-client</groupId>
        <artifactId>google-http-client-jackson2</artifactId>
        <version>1.33.0</version>
    </dependency>
    <dependency>
        <groupId>com.google.apis</groupId>
        <artifactId>google-api-services-oauth2</artifactId>
        <version>v2-rev151-1.25.0</version>
    </dependency>
    <dependency>
        <groupId>es.moki.ratelimitj</groupId>
        <artifactId>ratelimitj-redis</artifactId>
        <version>0.6.0</version>
    </dependency>
    <dependency>
        <groupId>com.auth0</groupId>
        <artifactId>java-jwt</artifactId>
        <version>3.3.0</version>
    </dependency>
    <dependency>
        <groupId>com.auth0</groupId>
        <artifactId>jwks-rsa</artifactId>
        <version>0.9.0</version>
    </dependency>
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>s3</artifactId>
        <version>2.10.25</version>
    </dependency>
</dependencies>

<properties>
    <java.version>1.8</java.version>
</properties>


<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
    </plugins>
</build>

[efe-selcuk]

Thanks. It would take a bit of time to ramp up with maven and tomcat to replicate this. If you have a minimal sample, that'll help quite a bit.

In general, I'll ask for a few things:

• Which Envoy image are you running?
• Can you share your virtual node and virtual service configurations? Also curling :9901/config_dump on the envoys to grab the configs would be great.
• Are those logs verbatim including the ${MESH_NAME} or did you replace those yourself?
• Can you provide more of the headers from both the upstream and downstream?

That being said, from what I can tell Envoy wouldn't be changing the host header for HTTP. If it's not set by its local application, it'll actually respond back with an error, least for HTTP/1.1. Though it may for gRPC.

Also, please clarify ingress/egress here. If we're assuming something like:

App1 <-> Envoy1 <--------> Envoy2 <-> App2

Where the apps are tomcat servers, in which app are you seeing this error? And in which direction is the request going?

[dfezzie]

I have a repro of the problem at https://github.com/dfezzie/SpringBootWithAppMesh

I'm working on figuring out what exactly is going on. I'll update when I have a bit more information

[sybeck2k]

I'm seeing this too with a Spring Boot app. Adding to it, this makes impossible to get the health check to pass for App Mesh, since Envoy is querying the health check using a host name that contains underscores which Tomcat considers invalid, and thus returns 400.

[dfezzie]

Hey all, I’ve identified what the issue is. When access logging is turned on, you can see health checks use the cluster name as the host header for the request. Our current health check configuration omits the host field. According to the envoy documentation:

If left empty (default value), the name of the cluster this health check is associated with will be used

This issue also affects GRPC health checks, which has a similar field called authority. Instead of leaving the host field blank, we will use the Service Discovery field on the Virtual Node to populate the host field. For DNS Service Discovery, we will use the hostname field and for Cloud Map Service Discovery, we will use the serviceName.namespaceName. The combination of serviceName.namespaceName is the DNS Name for the service.

We are working on the fix and will update when the fix is present in all regions

[dfezzie]

The fix is present in all regions. Going to close out this issue 👍