Jetty.project: NPE resulting from bad JEP 238 MultiReleaseJarFile structure

[phax]

In my current setup (Jetty 9.4.7 and JDK 1.8.0_144 64 Bit on Win 64) I get an error starting up Jetty, with annotation scanning enabled:

[qtp1889057031-16]  -- org.eclipse.jetty.annotations.ClassInheritanceHandler.handle(ClassInheritanceHandler.java:65)
java.lang.NullPointerException: null
    at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[?:1.8.0_144]
    at org.eclipse.jetty.annotations.ClassInheritanceHandler.addToInheritanceMap(ClassInheritanceHandler.java:73) ~[jetty-annotations-9.4.7.v20170914.jar:9.4.7.v20170914]
    at org.eclipse.jetty.annotations.ClassInheritanceHandler.handle(ClassInheritanceHandler.java:59) [jetty-annotations-9.4.7.v20170914.jar:9.4.7.v20170914]
    at org.eclipse.jetty.annotations.AnnotationParser$MyClassVisitor.visit(AnnotationParser.java:496) [jetty-annotations-9.4.7.v20170914.jar:9.4.7.v20170914]
    at org.objectweb.asm.ClassReader.accept(Unknown Source) [asm-5.1.jar:5.1]
    at org.objectweb.asm.ClassReader.accept(Unknown Source) [asm-5.1.jar:5.1]
    at org.eclipse.jetty.annotations.AnnotationParser.scanClass(AnnotationParser.java:980) [jetty-annotations-9.4.7.v20170914.jar:9.4.7.v20170914]
    at org.eclipse.jetty.annotations.AnnotationParser.parseJarEntry(AnnotationParser.java:960) [jetty-annotations-9.4.7.v20170914.jar:9.4.7.v20170914]
    at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:907) [jetty-annotations-9.4.7.v20170914.jar:9.4.7.v20170914]
    at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:857) [jetty-annotations-9.4.7.v20170914.jar:9.4.7.v20170914]
    at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:159) [jetty-annotations-9.4.7.v20170914.jar:9.4.7.v20170914]
    at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:462) [jetty-annotations-9.4.7.v20170914.jar:9.4.7.v20170914]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:679) [jetty-util-9.4.7.v20170914.jar:9.4.7.v20170914]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:597) [jetty-util-9.4.7.v20170914.jar:9.4.7.v20170914]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

The reason is in the following method in class ClassInheritanceHandler:

    public void handle(ClassInfo classInfo)
    {
        try
        {
            for (int i=0; classInfo.getInterfaces() != null && i < classInfo.getInterfaces().length;i++)
            {
                addToInheritanceMap(classInfo.getInterfaces()[i], classInfo.getClassName());
                //_inheritanceMap.add (classInfo.getInterfaces()[i], classInfo.getClassName());
            }
            //To save memory, we don't record classes that only extend Object, as that can be assumed
            if (!"java.lang.Object".equals(classInfo.getSuperName()))
            {
// -- we should not enter this branch if getSuperName returns null
                addToInheritanceMap(classInfo.getSuperName(), classInfo.getClassName());
                //_inheritanceMap.add(classInfo.getSuperName(), classInfo.getClassName());
            }
        }
        catch (Exception e)
        {
            LOG.warn(e);
        }  
    }

because classInfo.getSuperName() for the class info of class java.lang.Object returns null. See the comment above.

Update: the same error occurrs with Jetty 9.4.6 - so it might be a problem with the JDK 1.8.0_144 version...
Update 2: in v9.4.6 the application starts up anyway, in v9.4.7 the Jetty bootstrapping fails

[janbartel]

@phax, can you supply a reproduction test case? I can't understand why you would be scanning the java.lang.Object class itself as this is not in a jar that is normally subject to scanning.

[phax]

It happens as part of a bigger project.
As this annotation scanning happens in parallel I'm having a hard time to provide some more meaningful insights. I can try to build a mini example using my personal dependencies and we can hope that it crashes...

[janbartel]

Yes please. Can you also tell me what the org.eclipse.jetty.server.webapp.ContainerIncludeJarPattern is set to? And maybe the org.eclipse.jetty.server.webapp.WebInfIncludeJarPattern for good measure.

Unless the ContainerIncludeJarPattern is including every single jar on the container classpath all the way to the root loader, I can't see how you could wind up scanning java.lang.Object. Moreover, java.lang.Object doesn't have any interfaces, so again, I can't see how we're getting to that snippet of code you posted.

[phax]

I'm trying to narrow it down. I think it might have something to do with Eclipse resolving Maven SNAPSHOTs to open projects - so having a directory and not a JAR file as a dependency. But this is work in progress.

Here are settings that may be of interest:

    final WebAppContext aWebAppCtx = new WebAppContext ();
...
      aWebAppCtx.setParentLoaderPriority (true);
      aWebAppCtx.setThrowUnavailableOnStartupException (true);
      // http://www.eclipse.org/jetty/documentation/9.3.x/configuring-webapps.html#container-include-jar-pattern
      // https://github.com/eclipse/jetty.project/issues/680
      aWebAppCtx.setAttribute (WebInfConfiguration.CONTAINER_JAR_PATTERN, ".*\\.jar$|.*/classes/.*");

[phax]

I can't reproduce it in a software easily, but I have a stacktrace where it happens. If you tell me what information is of interest for you, I'm happy to let you know:

ParserTask running: org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask@67f936b5
AnnotationParser.parse: Resource=file:///C:/Program%20Files/Java/jdk1.8.0_144/jre/lib/rt.jar

[magnayn]

I am getting the same, but with a slightly different stacktrace:

this is from my 'test' server that I run in my IDE. My uberjar which contains the same version of jetty (but is slightly different jetty startup code) seems to work.

[qtp1057507014-24] WARN  o.e.j.a.ClassInheritanceHandler -  
java.lang.NullPointerException: null
    at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[na:1.8.0_73]
    at org.eclipse.jetty.annotations.ClassInheritanceHandler.addToInheritanceMap(ClassInheritanceHandler.java:72) ~[jetty-annotations-9.3.9.v20160517.jar:9.3.9.v20160517]
    at org.eclipse.jetty.annotations.ClassInheritanceHandler.handle(ClassInheritanceHandler.java:58) ~[jetty-annotations-9.3.9.v20160517.jar:9.3.9.v20160517]
    at org.eclipse.jetty.annotations.AnnotationParser$MyClassVisitor.visit(AnnotationParser.java:476) [jetty-annotations-9.3.9.v20160517.jar:9.3.9.v20160517]
    at org.objectweb.asm.ClassReader.accept(Unknown Source) [asm-5.0.1.jar:5.0.1]
    at org.objectweb.asm.ClassReader.accept(Unknown Source) [asm-5.0.1.jar:5.0.1]
    at org.eclipse.jetty.annotations.AnnotationParser.scanClass(AnnotationParser.java:1004) [jetty-annotations-9.3.9.v20160517.jar:9.3.9.v20160517]
    at org.eclipse.jetty.annotations.AnnotationParser.parseJarEntry(AnnotationParser.java:984) [jetty-annotations-9.3.9.v20160517.jar:9.3.9.v20160517]
    at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:933) [jetty-annotations-9.3.9.v20160517.jar:9.3.9.v20160517]
    at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:851) [jetty-annotations-9.3.9.v20160517.jar:9.3.9.v20160517]
    at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:163) [jetty-annotations-9.3.9.v20160517.jar:9.3.9.v20160517]
    at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:546) [jetty-annotations-9.3.9.v20160517.jar:9.3.9.v20160517]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_73]

[phax]

One difference is the asm version. You have 5.0.1, I have 5.1 (the latest 5.x version is 5.2)....
And it seems to be independent of the Java version (1.8.0_144 vs 1.8.0_73)
I still have the active stacktrace - if any of the Jetty developers need a variable content or so - let me know

[janbartel]

Well I give up: I can't reproduce it on linux running with your settings either in the distro, embedded or embedded running in eclipse. Must be something specific to windows.

In any case, I've checked in a change to ClassInheritanceHandler.

I'd appreciate it if you could build jetty locally and test the change.

[joakime]

@janbartel here's a testcase that cause a NPE (before your latest commit) in AnnotationParser.parse()

    @Test
    public void testNpeInJavaRuntime() throws Exception
    {
        URI stringLocation = TypeUtil.getLocationOfClass(String.class);
        assumeThat(stringLocation, notNullValue()); // skip on odd JVMs
        AnnotationParser parser = new AnnotationParser();
        Set<Handler> emptySet = Collections.emptySet();
        parser.parse(emptySet, stringLocation);
        // Should throw no exceptions
    }

[janbartel]

@phax have you been able to test the change I committed?

[phax]

Is there a public Maven SNAPSHOT repository where I can retrieve 9.4.8-SNAPSHOT? It's obviously not on the oss.sonatype repo. Thx

[olamy]

You can use

    <repository>
      <id>jetty.snapshot</id>
      <url>https://oss.sonatype.org/content/repositories/jetty-snapshots/</url>
      <releases>
        <enabled>false</enabled>
      </releases>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
    </repository>

Please note snapshots are build & deployed once a day by this Jenkins job: https://jenkins.webtide.net/job/nightlies/job/jetty-9.4-snapshot/

[phax]

I tried the latest SNAPSHOT but run again in problem with multi release JAR :(

java.lang.NullPointerException: null
    at org.eclipse.jetty.util.MultiReleaseJarFile.<init>(MultiReleaseJarFile.java:92) ~[jetty-util-9.4.8-SNAPSHOT.jar:9.4.8-SNAPSHOT]
    at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:880) ~[jetty-annotations-9.4.8-SNAPSHOT.jar:9.4.8-SNAPSHOT]
    at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:846) ~[jetty-annotations-9.4.8-SNAPSHOT.jar:9.4.8-SNAPSHOT]
    at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:160) ~[jetty-annotations-9.4.8-SNAPSHOT.jar:9.4.8-SNAPSHOT]
    at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:467) ~[jetty-annotations-9.4.8-SNAPSHOT.jar:9.4.8-SNAPSHOT]

I was trying with Log4J 2.9.1 (as in the original multi jar issue #1797 ) - you may wanna create a separate issue from this...

The problem is that outer is null because this is reached for rt.jar (1.8.0_144, Win 10, 64 Bit Java)

[gregw]

@phax
It looks like that NPE can occur only if a jar contains an inner class (eg org/example/Foo$Bar but not it's outer class org/example/Foo.class.

Does that sound plausible for your test?

Regardless, I will fix this issue by ignoring such classes. They may have worked in the past with their outer classes being in different jars, but no longer in a MR jar world!

[phax]

Works with todays SNAPSHOT, so the issue can be closed

[nadavwe]

Hi, I get a similar exception on 9.3.22.v20171030, after upgrading from 9.3.17.v20170317.
could it be that this was fixed on the 9.4 versions but got into the 9.3 versions as well?
Thanks!

[janbartel]

@nadavwe can you clarify if you're asking about the original reported error on this issue (NPE in ClassInheritanceHandler), or if you're referring to the subsequent error reported by @phax (NPE in MultiReleaseJarFile)?

[gregw]

@nadavwe and please post your exact exception, as "similar" is not good enough to debug with!

[nadavwe]

hey @gregw, @janbartel sorry for this, it was a little silly.
this was on 9.3.22.v20171030.

MultiException[java.lang.NullPointerException, java.lang.NullPointerException, java.lang.NullPointerException]
    at org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations(AnnotationConfiguration.java:539)
    at org.eclipse.jetty.annotations.AnnotationConfiguration.configure(AnnotationConfiguration.java:448)
    at org.eclipse.jetty.webapp.WebAppContext.configure(WebAppContext.java:496)
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1363)
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778)
    at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:522)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.handler.StatisticsHandler.doStart(StatisticsHandler.java:252)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
    at org.eclipse.jetty.server.Server.start(Server.java:422)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.Server.doStart(Server.java:389)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at com.wix.bootstrap.jetty.JettyBasedServer.com$wix$bootstrap$jetty$JettyBasedServer$$startServer(JettyBasedServer.scala:157)
    at com.wix.bootstrap.jetty.JettyBasedServer$$anonfun$start$1.apply$mcV$sp(JettyBasedServer.scala:39)
    at com.wix.bootstrap.jetty.JettyBasedServer.withServerStopOnException(JettyBasedServer.scala:145)
    at com.wix.bootstrap.jetty.JettyBasedServer.start(JettyBasedServer.scala:38)
    at com.wix.bootstrap.jetty.EmbeddedWar$.main(EmbeddedWar.scala:22)
    at com.wix.bootstrap.jetty.EmbeddedWar.main(EmbeddedWar.scala)
Suppressed:
    |java.lang.NullPointerException
    |   at org.eclipse.jetty.util.MultiReleaseJarFile.<init>(MultiReleaseJarFile.java:92)
    |   at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:926)
    |   at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:890)
    |   at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:164)
    |   at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:551)
    |   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
    |   at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
    |   at java.lang.Thread.run(Thread.java:745)
Suppressed:
    |java.lang.NullPointerException
    |   at org.eclipse.jetty.util.MultiReleaseJarFile.<init>(MultiReleaseJarFile.java:92)
    |   at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:926)
    |   at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:890)
    |   at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:164)
    |   at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:551)
    |   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
    |   at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
    |   at java.lang.Thread.run(Thread.java:745)
Caused by:
java.lang.NullPointerException
    at org.eclipse.jetty.util.MultiReleaseJarFile.<init>(MultiReleaseJarFile.java:92)
    at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:926)
    at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:890)
    at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:164)
    at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:551)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
    at java.lang.Thread.run(Thread.java:745)

[janbartel]

@nadavwe 9.3.22 was released on 30th October, and the two relevant commits were made on 2nd (https://github.com/eclipse/jetty.project/commit/6e7560acd1704903be6142a001a5fc6f6ba8547d) and 8th
(https://github.com/eclipse/jetty.project/commit/b1d5fea96cd9e8d40fb5446501e2b18cb9971e7b) November. So you need us to make release 9.3.23 :) Stay tuned.

[nadavwe]

Thanks for the reply. =)
can you confirm that 9.3.22 is effected by this bug as well?

[phax]

@nadavwe That is clearly a yes

[jeremyhill-up]

Same problem on JDK 1.8.0_131 and jetty-9.2.23.v20171218 and log4j2 2.10.0

[INFO] jetty-9.2.23.v20171218
[WARNING]
java.lang.NullPointerException
        at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
        at org.eclipse.jetty.annotations.ClassInheritanceHandler.addToInheritanceMap(ClassInheritanceHandler.java:72)
        at org.eclipse.jetty.annotations.ClassInheritanceHandler.handle(ClassInheritanceHandler.java:58)
        at org.eclipse.jetty.annotations.AnnotationParser$MyClassVisitor.visit(AnnotationParser.java:480)
        at org.objectweb.asm.ClassReader.accept(ClassReader.java:621)
        at org.objectweb.asm.ClassReader.accept(ClassReader.java:500)
        at org.eclipse.jetty.annotations.AnnotationParser.scanClass(AnnotationParser.java:974)
        at org.eclipse.jetty.annotations.AnnotationParser.parseJarEntry(AnnotationParser.java:956)
        at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:909)
        at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:831)
        at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:164)
        at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:549)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)
java.lang.RuntimeException: Error scanning entry module-info.class from jar file:(snip)/m2/repository/org/apache/logging/log4j/log4j-api/2.10.0/log4j-api-2.10.0.jar
        at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:913)
        at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:831)
        at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:164)
        at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:549)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException
        at org.objectweb.asm.ClassVisitor.visitModule(ClassVisitor.java:148)
        at org.objectweb.asm.ClassReader.readModule(ClassReader.java:731)
        at org.objectweb.asm.ClassReader.accept(ClassReader.java:632)
        at org.objectweb.asm.ClassReader.accept(ClassReader.java:500)
        at org.eclipse.jetty.annotations.AnnotationParser.scanClass(AnnotationParser.java:974)
        at org.eclipse.jetty.annotations.AnnotationParser.parseJarEntry(AnnotationParser.java:956)
        at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:909)
        ... 6 more

[joakime]

Jetty 9.2 has no support for JEP 238 and Java 9, sorry.

[vinayapp]

Thanks. Similar issue here: https://stackoverflow.com/questions/49492583/sprint-boot-web-application-running-on-google-app-engine-throws-jetty-exceptio . So then how can we run a Spring Boot application (.war) with Gradle build on App Engine Standard.

[joakime]

@vinayapp search your WEB-INF/lib/*.jar files for entries that have module-info.class files or start with META-INF/versions/* and downgrade those artifacts to ones that don't use the Java 9 features.

[joakime]

@vinayapp or alternatively, look at your exceptions, they'll tell you which artifacts are incompatible for your Spring Boot and GAE combination.

Example from @jeremyhill-up

java.lang.RuntimeException: Error scanning entry module-info.class from jar file:(snip)/m2/repository/org/apache/logging/log4j/log4j-api/2.10.0/log4j-api-2.10.0.jar
        at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:913)

[vinayapp]

Sure. Will check on this.