Payara: HTTP/2 ServerPush not working in Payara Micro 5.182 due to ALPN not being turned on (for some reason)

[progaddict]

# Description #

The dockerized payara/micro:5.182 does not seem to support HTTP/2 and in particular the PushBuilder feature from the Servlet 4.0 spec ( or maybe I do not know "hot to cook it" :smiley: )

Expected Outcome

When I deploy locally a simple single-servlet-app (see details below in the "steps to reproduce" section), which pushes an image along with a simple html page referencing it, and then go to https://localhost:8443/ then I expect to see a single HTTP/2 request:

Current Outcome

However, instead I see two HTTP 1.1 requests:

Steps to reproduce

1. clone the repository:
   git clone --depth=1 [email protected]:progaddict/some-java-ee-stuff.git
2. go to the sample with the single-servlet-app:
   cd some-java-ee-stuff/servlet-4.0-server-push
3. build it:
   mvn clean install
   (it should also build the corresponding docker image http2-servlet-payara-micro. so make sure you have docker installed on your system.)
4. go to the dockerized payara project:
   cd servlet-4.0-server-push-payara-micro
5. start the previously built container:
   mvn docker:start
6. when container starts, go to https://localhost:8443/. You'll need to accept the certificate exception. Go to the network sniffer tab (e.g. F12 in FireFox) and refresh the page (Ctrl+F5).
7. Observe that there are two HTTP 1.1 requests instead of a single HTTP/2.
8. Read the container's logs:
   docker logs <ID or name of the container that runs the app>
9. And observe that the HTTP/2 was not enabled due to the switched off ALPN:

[2018-08-19T12:17:42.494+0000] [] [INFO] [] [fish.payara.micro.boot.runtime.PayaraMicroRuntimeBuilder] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1534681062494] [levelValue: 800] Built Payara Micro Runtime
[2018-08-19T12:17:43.128+0000] [] [INFO] [NCLS-CORE-00046] [javax.enterprise.system.core] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1534681063128] [levelValue: 800] Cannot find javadb client jar file, derby jdbc driver will not be available by default.
[2018-08-19T12:17:44.465+0000] [] [INFO] [] [org.glassfish.ha.store.spi.BackingStoreFactoryRegistry] [tid: _ThreadID=19 _ThreadName=RunLevelControllerThread-1534681063861] [timeMillis: 1534681064465] [levelValue: 800] Registered fish.payara.ha.hazelcast.store.HazelcastBackingStoreFactoryProxy for persistence-type = hazelcast in BackingStoreFactoryRegistry
[2018-08-19T12:17:44.790+0000] [] [INFO] [NCLS-CORE-00101] [javax.enterprise.system.core] [tid: _ThreadID=21 _ThreadName=RunLevelControllerThread-1534681063867] [timeMillis: 1534681064790] [levelValue: 800] Network Listener http-listener started in: 62ms - bound to [/0.0.0.0:8080]
[2018-08-19T12:17:44.846+0000] [] [WARNING] [] [org.glassfish.grizzly.http2.Http2AddOn] [tid: _ThreadID=21 _ThreadName=RunLevelControllerThread-1534681063867] [timeMillis: 1534681064846] [levelValue: 900] TLS ALPN (Application-Layer Protocol Negotiation) support is not available. HTTP/2 support will not be enabled.
[2018-08-19T12:17:44.888+0000] [] [INFO] [NCLS-CORE-00101] [javax.enterprise.system.core] [tid: _ThreadID=21 _ThreadName=RunLevelControllerThread-1534681063867] [timeMillis: 1534681064888] [levelValue: 800] Network Listener https-listener started in: 13ms - bound to [/0.0.0.0:8181]
[2018-08-19T12:17:44.890+0000] [] [INFO] [NCLS-CORE-00087] [javax.enterprise.system.core] [tid: _ThreadID=21 _ThreadName=RunLevelControllerThread-1534681063867] [timeMillis: 1534681064890] [levelValue: 800] Grizzly 2.4.3 started in: 774ms - bound to [http-listener:8080, https-listener:8181]
[2018-08-19T12:17:45.107+0000] [] [INFO] [NCLS-CORE-00017] [javax.enterprise.system.core] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1534681065107] [levelValue: 800] Payara Micro  5.182 #badassmicrofish (303) startup time : Embedded (1,337ms), startup services(1,250ms), total(2,587ms)
[2018-08-19T12:17:45.216+0000] [] [INFO] [NCLS-JMX-00006] [javax.enterprise.system.jmx] [tid: _ThreadID=69 _ThreadName=Thread-8] [timeMillis: 1534681065216] [levelValue: 800] JMXStartupService has disabled JMXConnector system
...

note: the WARNING log message above says "TLS ALPN (Application-Layer Protocol Negotiation) support is not available. HTTP/2 support will not be enabled."

Samples

https://github.com/progaddict/some-java-ee-stuff/tree/0d5406e9fb4d49ba83ba5d72f17043c3ab3033bb/servlet-4.0-server-push

Context

I'm trying the PushBuilder feature from the Servlet 4.0 specification. It works with Tomcat 9 (see https://github.com/progaddict/some-java-ee-stuff/tree/master/servlet-4.0-server-push/servlet-4.0-server-push-tomcat) but does not work with payara-micro 5.182.

Environment

• Payara Version: 5.182
• Edition: Micro
• JDK Version: the version that is installed in the payara/micro:5.182 docker image
• Operating System: Linux (in a docker container)

[lprimak]

Http2 server push is broken in grizzly right now. We are working to fix that

[smillidge]

The docker image isn't configured to run HTTP/2. As it just runs java -jar payara-micro.jar. You need to add the alpn jar to the boot classpath on the command line to enable HTTP/2 using the command line option

-Xbootclasspath/p:grizzly-npn-bootstrap.jar

You can find the Payara Server npn jar here https://search.maven.org/artifact/org.glassfish.grizzly/grizzly-npn-bootstrap/1.8/jar or from inside the payara-micro.jar.

[progaddict]

thanks a lot for the quick reply!

I've adjusted my deployment:
https://github.com/progaddict/some-java-ee-stuff/commit/e604dfbe8ec379cff88c60d9baa056a379463aac

now it runs the following command inside the container:
java -Xbootclasspath/p:/my-app/grizzly-npn-bootstrap.jar -jar /opt/payara/payara-micro.jar --sslPort 8181 --deploy /my-app/ROOT.war

there are no log warning during statup anymore. however, there's a runtime error now:

[2018-08-19T19:07:07.724+0000] [] [WARNING] [] [org.glassfish.grizzly.filterchain.DefaultFilterChain] [tid: _ThreadID=48 _ThreadName=http-thread-pool::https-listener(1)] [timeMillis: 1534705627724] [levelValue: 900] [[
  GRIZZLY0013: Exception during FilterChain execution
java.lang.NoClassDefFoundError: sun/security/ssl/EllipticCurvesExtension
    at sun.security.ssl.Handshaker.getActiveProtocols(Handshaker.java:793)
    at sun.security.ssl.Handshaker.activate(Handshaker.java:549)
    at sun.security.ssl.SSLEngineImpl.kickstartHandshake(SSLEngineImpl.java:744)
    at sun.security.ssl.SSLEngineImpl.beginHandshake(SSLEngineImpl.java:771)
    at org.glassfish.grizzly.ssl.SSLBaseFilter$SSLTransportFilterWrapper.handleRead(SSLBaseFilter.java:1113)
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88)
    at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53)
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:524)
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:94)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:33)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:114)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549)
    at java.lang.Thread.run(Thread.java:748)
]]

googling led me to the (possibly) relevant set of issues:

• #2513
• #2958
• https://github.com/eclipse-ee4j/grizzly-npn/issues/4

however, all these issues are closed/resolved. in particular, #2513 suggests that the JDK version might be incompatible with payara-micro 5.x. but I'm running payara-micro inside a container which I've built from the payara/micro:5.182 and therefore whatever JDK is used inside the container, it has been tested with the 5.182, right?

[progaddict]

the container's java has the following version:

openjdk version "1.8.0_151"
OpenJDK Runtime Environment (IcedTea 3.6.0) (Alpine 8.151.12-r0)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

[progaddict]

oh! I think I got it!
the container has actually an old version of the JDK.
in the upcoming week I'll try to rebuild the container using a newer version of JDK and I'll post the results here

[progaddict]

I've adjusted the Dockerfile to use the openjdk:8u171-jdk-alpine:
https://github.com/progaddict/some-java-ee-stuff/commit/5132ec14999e131c738a29127a31b29a618d628f

And now the dockerized app works as expected: now I see exactly one HTTP/2 request.

It would be really nice, if you could update your payara/micro:5.182 to use a newer JDK version.

[smillidge]

Please raise the issue on the payara docker repo in Github.