Presto: Java 11 causes Error injecting constructor, java.io.IOException: Can not attach to current VM

Created on 12 May 2019  路  13Comments  路  Source: prestosql/presto

Getting the following error when trying to run Presto 310 with Java 11. I also found the same issue in the prestosql slack channel https://prestosql.slack.com/archives/CFLB9AMBN/p1554760462185400.

2019-05-12T04:24:51.779Z    ERROR   main    io.prestosql.server.PrestoServer    Unable to create injector, see the following errors:

1) Error injecting constructor, java.io.IOException: Can not attach to current VM
  at io.airlift.jmx.JmxAgent9.<init>(JmxAgent9.java:41)
  at io.airlift.jmx.JmxModule.configure(JmxModule.java:55)
  while locating io.airlift.jmx.JmxAgent9
  while locating io.airlift.jmx.JmxAgent
    for the 1st parameter of io.airlift.jmx.JmxModule$JmxAnnouncementProvider.setJmxAgent(JmxModule.java:68)
  while locating io.airlift.jmx.JmxModule$JmxAnnouncementProvider
  while locating io.airlift.discovery.client.ServiceAnnouncement annotated with @com.google.inject.internal.Element(setName=,uniqueId=331, type=MULTIBINDER, keyType=)
  while locating java.util.Set<io.airlift.discovery.client.ServiceAnnouncement>
    for the 2nd parameter of io.airlift.discovery.client.Announcer.<init>(Announcer.java:68)
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:64)
  while locating io.airlift.discovery.client.Announcer
    for the 2nd parameter of io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  at io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  while locating io.airlift.discovery.client.MergingServiceSelectorFactory
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:73)
  while locating io.airlift.discovery.client.ServiceSelectorFactory
    for the 1st parameter of io.airlift.discovery.client.ServiceSelectorProvider.setServiceSelectorFactory(ServiceSelectorProvider.java:50)
  at io.airlift.event.client.HttpEventModule.configure(HttpEventModule.java:43)

2) Error injecting constructor, java.io.IOException: Can not attach to current VM
  at io.airlift.jmx.JmxAgent9.<init>(JmxAgent9.java:41)
  at io.airlift.jmx.JmxModule.configure(JmxModule.java:55)
  while locating io.airlift.jmx.JmxAgent9
  while locating io.airlift.jmx.JmxAgent
    for the 1st parameter of io.airlift.jmx.JmxModule$JmxAnnouncementProvider.setJmxAgent(JmxModule.java:68)
  while locating io.airlift.jmx.JmxModule$JmxAnnouncementProvider
  while locating io.airlift.discovery.client.ServiceAnnouncement annotated with @com.google.inject.internal.Element(setName=,uniqueId=331, type=MULTIBINDER, keyType=)
  while locating java.util.Set<io.airlift.discovery.client.ServiceAnnouncement>
    for the 2nd parameter of io.airlift.discovery.client.Announcer.<init>(Announcer.java:68)
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:64)
  while locating io.airlift.discovery.client.Announcer
    for the 2nd parameter of io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  at io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  while locating io.airlift.discovery.client.MergingServiceSelectorFactory
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:73)
  while locating io.airlift.discovery.client.ServiceSelectorFactory
    for the 1st parameter of io.airlift.discovery.client.ServiceSelectorProvider.setServiceSelectorFactory(ServiceSelectorProvider.java:50)
  at io.prestosql.server.ServerMainModule.setup(ServerMainModule.java:232)

2 errors
com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Error injecting constructor, java.io.IOException: Can not attach to current VM
  at io.airlift.jmx.JmxAgent9.<init>(JmxAgent9.java:41)
  at io.airlift.jmx.JmxModule.configure(JmxModule.java:55)
  while locating io.airlift.jmx.JmxAgent9
  while locating io.airlift.jmx.JmxAgent
    for the 1st parameter of io.airlift.jmx.JmxModule$JmxAnnouncementProvider.setJmxAgent(JmxModule.java:68)
  while locating io.airlift.jmx.JmxModule$JmxAnnouncementProvider
  while locating io.airlift.discovery.client.ServiceAnnouncement annotated with @com.google.inject.internal.Element(setName=,uniqueId=331, type=MULTIBINDER, keyType=)
  while locating java.util.Set<io.airlift.discovery.client.ServiceAnnouncement>
    for the 2nd parameter of io.airlift.discovery.client.Announcer.<init>(Announcer.java:68)
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:64)
  while locating io.airlift.discovery.client.Announcer
    for the 2nd parameter of io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  at io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  while locating io.airlift.discovery.client.MergingServiceSelectorFactory
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:73)
  while locating io.airlift.discovery.client.ServiceSelectorFactory
    for the 1st parameter of io.airlift.discovery.client.ServiceSelectorProvider.setServiceSelectorFactory(ServiceSelectorProvider.java:50)
  at io.airlift.event.client.HttpEventModule.configure(HttpEventModule.java:43)

2) Error injecting constructor, java.io.IOException: Can not attach to current VM
  at io.airlift.jmx.JmxAgent9.<init>(JmxAgent9.java:41)
  at io.airlift.jmx.JmxModule.configure(JmxModule.java:55)
  while locating io.airlift.jmx.JmxAgent9
  while locating io.airlift.jmx.JmxAgent
    for the 1st parameter of io.airlift.jmx.JmxModule$JmxAnnouncementProvider.setJmxAgent(JmxModule.java:68)
  while locating io.airlift.jmx.JmxModule$JmxAnnouncementProvider
  while locating io.airlift.discovery.client.ServiceAnnouncement annotated with @com.google.inject.internal.Element(setName=,uniqueId=331, type=MULTIBINDER, keyType=)
  while locating java.util.Set<io.airlift.discovery.client.ServiceAnnouncement>
    for the 2nd parameter of io.airlift.discovery.client.Announcer.<init>(Announcer.java:68)
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:64)
  while locating io.airlift.discovery.client.Announcer
    for the 2nd parameter of io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  at io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  while locating io.airlift.discovery.client.MergingServiceSelectorFactory
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:73)
  while locating io.airlift.discovery.client.ServiceSelectorFactory
    for the 1st parameter of io.airlift.discovery.client.ServiceSelectorProvider.setServiceSelectorFactory(ServiceSelectorProvider.java:50)
  at io.prestosql.server.ServerMainModule.setup(ServerMainModule.java:232)

2 errors
    at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:178)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
    at com.google.inject.Guice.createInjector(Guice.java:87)
    at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:240)
    at io.prestosql.server.PrestoServer.run(PrestoServer.java:120)
    at io.prestosql.server.PrestoServer.main(PrestoServer.java:70)
Caused by: java.io.IOException: Can not attach to current VM
    at jdk.attach/sun.tools.attach.HotSpotVirtualMachine.<init>(HotSpotVirtualMachine.java:75)
    at jdk.attach/sun.tools.attach.VirtualMachineImpl.<init>(VirtualMachineImpl.java:57)
    at jdk.attach/sun.tools.attach.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:58)
    at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:207)
    at io.airlift.jmx.JmxAgent9.<init>(JmxAgent9.java:56)
    at io.airlift.jmx.JmxAgent9$$FastClassByGuice$$52a284ef.newInstance(<generated>)
    at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
    at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32)
    at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:98)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
    at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
    at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
    at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
    at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:82)
    at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:147)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:124)
    at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32)
    at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:98)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
    at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
    at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
    at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:60)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
    at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:198)
    at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:151)
    at com.google.inject.internal.InternalProviderInstanceBindingImpl$Factory$1.call(InternalProviderInstanceBindingImpl.java:120)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
    at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
    at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
    at com.google.inject.internal.InternalProviderInstanceBindingImpl$Factory.get(InternalProviderInstanceBindingImpl.java:115)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
    at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32)
    at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:98)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
    at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
    at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
    at com.google.inject.internal.ProviderMethod.doProvision(ProviderMethod.java:173)
    at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.provision(InternalProviderInstanceBindingImpl.java:185)
    at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.access$300(InternalProviderInstanceBindingImpl.java:139)
    at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory$1.call(InternalProviderInstanceBindingImpl.java:169)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
    at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
    at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
    at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.get(InternalProviderInstanceBindingImpl.java:164)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
    at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
    at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:82)
    at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:147)
    at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:101)
    at com.google.inject.internal.Initializer$InjectableReference.get(Initializer.java:245)
    at com.google.inject.internal.Initializer.injectAll(Initializer.java:140)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:176)
    ... 5 more
picture chancez

Most helpful comment

Try adding this to etc/jvm.config:

-Djdk.attach.allowAttachSelf=true

If that's the problem, we should add that to the deployment instructions.

picture electrum on 12 May 2019
👍5

All 13 comments

Try adding this to etc/jvm.config:

-Djdk.attach.allowAttachSelf=true

If that's the problem, we should add that to the deployment instructions.

electrum picture electrum on 12 May 2019
👍5

So I tried that, and now I'm getting 

1) Error injecting constructor, com.sun.tools.attach.AttachOperationFailedException: java.lang.RuntimeException: Invalid agent state: Agent already started

Note, I am running with 

    -javaagent:/opt/jmx_exporter/jmx_exporter.jar=8082:/opt/jmx_exporter/config/config.yml

using https://github.com/prometheus/jmx_exporter

chancez picture chancez on 12 May 2019

Trying with a minimal set of flags to figure out if it's anything specific.

chancez picture chancez on 12 May 2019

It seems like these are the minimum properties needed to cause this problem:

    -Dcom.sun.management.jmxremote
    -Dcom.sun.management.jmxremote.port=8081
    -Dcom.sun.management.jmxremote.authenticate=false
    -Djdk.attach.allowAttachSelf=true
chancez picture chancez on 12 May 2019

Without any extra jvm flags, eg:
config.properties:

http-server.http.port=8080
coordinator=true
discovery-server.enabled=true
discovery.uri=http://presto:8080
node-scheduler.include-coordinator=true

jvm.config:

it causes the original error, and with -Djdk.attach.allowAttachSelf=true it works. So I can't seem to enable JMX at all currently, and even without JMX enabled, the allowAttachSelf must be true currently.

chancez picture chancez on 12 May 2019

The problem is that the first three flags tell the JVM to start the agent, then the Airlift code in JmxAgent9 unconditionally tries to start the agent itself. The code for Java 8 in JmxAgent8 detects that it's already running. I'm guessing there were problems doing this detection for Java 9.

We either need to fix this detection, or add a config flag to skip loading the agent. This flag might need to go in Presto to conditionally add the Guice module JmxModule, since that module assumes the JMX agent is available (it publishes a JMX announcement in discovery).

electrum picture electrum on 12 May 2019

I see, so it shouldn't occur if I set the jmx options in presto only?

Also, what about jdk.attach.allowAttachSelf=true is that just kinda a default option needed since JMX is started unconditionally?

chancez picture chancez on 12 May 2019

jdk.attach.allowAttachSelf is what allows the Airlift JXM agent code to connect to the current JVM and start the JMX agent. It relaxes a security check in the JVM.

electrum picture electrum on 12 May 2019

I see, that makes sense.

So status update for anyone else who runs into these issues:

I was able to still use JMX by only using

    jmx.rmiserver.port=8081
    jmx.rmiregistry.port=8081

in config.properties, and

    -javaagent:/opt/jmx_exporter/jmx_exporter.jar=8082:/opt/jmx_exporter/config/config.yml
    # this fixes https://github.com/prestosql/presto/issues/752
    -Djdk.attach.allowAttachSelf=true

In jvm.properties.

All seems to be working, including my Prometheus JMX exporter.

Once you touch com.sun.management.jmxremote is when things stop working, for the reasons @electrum already mentioned.

chancez picture chancez on 12 May 2019
👍1

Thanks for helping us find this confusing scenario. I have a patch to add a hint about the self-attach flag and to skip starting the agent if it's already configured: https://github.com/airlift/airlift/pull/733

electrum picture electrum on 12 May 2019

This was fixed in Presto 312.

electrum picture electrum on 27 Jun 2019 
-Djdk.attach.allowAttachSelf=true

This solved the issue for me @electrum, if that's the case for most of us, please add it to deployment documentation.

codekaust picture codekaust on 1 Jun 2020

@codekaust thanks. This is already part of the docs -- https://prestosql.io/docs/current/installation/deployment.html#jvm-config
Please let me know if you see other places in docs which would require an update.

findepi picture findepi on 1 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

bill-warshaw picture bill-warshaw  路  4Comments
lxynov picture lxynov  路  4Comments
byungnam picture byungnam  路  4Comments
findepi picture findepi  路  3Comments
electrum picture electrum  路  5Comments