Quarkus: Infinispan can't be manually started with Quarkus 1.3.0

[burmanm]

Describe the bug
When launching the application that uses Infinispan directly (not through infinispan-embedded), the startup fails to loading metrics:

2020-03-17 13:35:10,563 ERROR [io.qua.application] (main) Failed to start application: java.lang.IllegalStateException: Display name differs from previous usage
    at io.smallrye.metrics.MetricsRegistryImpl.verifyMetadataEquality(MetricsRegistryImpl.java:203)
    at io.smallrye.metrics.MetricsRegistryImpl.register(MetricsRegistryImpl.java:151)
    at io.quarkus.smallrye.metrics.runtime.SmallRyeMetricsRecorder.garbageCollectionMetrics(SmallRyeMetricsRecorder.java:250)
    at io.quarkus.smallrye.metrics.runtime.SmallRyeMetricsRecorder.registerBaseMetrics(SmallRyeMetricsRecorder.java:110)
    at io.quarkus.deployment.steps.SmallRyeMetricsProcessor$registerBaseAndVendorMetrics18.deploy_0(SmallRyeMetricsProcessor$registerBaseAndVendorMetrics18.zig:65)
    at io.quarkus.deployment.steps.SmallRyeMetricsProcessor$registerBaseAndVendorMetrics18.deploy(SmallRyeMetricsProcessor$registerBaseAndVendorMetrics18.zig:36)
    at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:138)
    at io.quarkus.runtime.Application.start(Application.java:90)
    at io.quarkus.runtime.Application.run(Application.java:228)
    at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:41)

This is visible here because of #7907 that causes the displayName to differ from the one that Infinispan loaded, since Infinispan's new DefaultCacheManager(is) loads its own metrics before Quarkus:

DefaultMetadata{name='gc.total', type=counter, unit='none', reusable=true, description='Displays the total number of collections that have occurred. This attribute lists -1 if the collection count is undefined for this collector.', displayName='Garbage Collection Count'}

register:116, MetricsRegistryImpl (io.smallrye.metrics)
register:74, JmxRegistrar (io.smallrye.metrics.setup)
register:58, JmxRegistrar (io.smallrye.metrics.setup)
init:50, JmxRegistrar (io.smallrye.metrics.setup)
start:66, ApplicationMetricsRegistry (org.infinispan.metrics.impl)
start:41, CorePackageImpl$2 (org.infinispan.metrics.impl)
start:39, CorePackageImpl$2 (org.infinispan.metrics.impl)
invokeStart:587, BasicComponentRegistryImpl (org.infinispan.factories.impl)
doStartWrapper:578, BasicComponentRegistryImpl (org.infinispan.factories.impl)
startWrapper:547, BasicComponentRegistryImpl (org.infinispan.factories.impl)
access$700:30, BasicComponentRegistryImpl (org.infinispan.factories.impl)
running:770, BasicComponentRegistryImpl$ComponentWrapper (org.infinispan.factories.impl)
startDependencies:605, BasicComponentRegistryImpl (org.infinispan.factories.impl)
doStartWrapper:569, BasicComponentRegistryImpl (org.infinispan.factories.impl)
startWrapper:547, BasicComponentRegistryImpl (org.infinispan.factories.impl)
access$700:30, BasicComponentRegistryImpl (org.infinispan.factories.impl)
running:770, BasicComponentRegistryImpl$ComponentWrapper (org.infinispan.factories.impl)
startDependencies:605, BasicComponentRegistryImpl (org.infinispan.factories.impl)
doStartWrapper:569, BasicComponentRegistryImpl (org.infinispan.factories.impl)
startWrapper:547, BasicComponentRegistryImpl (org.infinispan.factories.impl)
access$700:30, BasicComponentRegistryImpl (org.infinispan.factories.impl)
running:770, BasicComponentRegistryImpl$ComponentWrapper (org.infinispan.factories.impl)
startDependencies:605, BasicComponentRegistryImpl (org.infinispan.factories.impl)
doStartWrapper:569, BasicComponentRegistryImpl (org.infinispan.factories.impl)
startWrapper:547, BasicComponentRegistryImpl (org.infinispan.factories.impl)
access$700:30, BasicComponentRegistryImpl (org.infinispan.factories.impl)
running:770, BasicComponentRegistryImpl$ComponentWrapper (org.infinispan.factories.impl)
registerComponentInternal:121, AbstractComponentRegistry (org.infinispan.factories)
registerComponent:110, AbstractComponentRegistry (org.infinispan.factories)
registerComponent:102, AbstractComponentRegistry (org.infinispan.factories)
cacheManagerStarting:106, LifecycleManager (org.infinispan.query.core.impl)
modulesManagerStarting:271, GlobalComponentRegistry (org.infinispan.factories)
access$000:65, GlobalComponentRegistry (org.infinispan.factories)
start:361, GlobalComponentRegistry$ModuleInitializer (org.infinispan.factories)
start:173, CorePackageImpl$11 (org.infinispan.factories)
start:171, CorePackageImpl$11 (org.infinispan.factories)
invokeStart:587, BasicComponentRegistryImpl (org.infinispan.factories.impl)
doStartWrapper:578, BasicComponentRegistryImpl (org.infinispan.factories.impl)
startWrapper:547, BasicComponentRegistryImpl (org.infinispan.factories.impl)
access$700:30, BasicComponentRegistryImpl (org.infinispan.factories.impl)
running:770, BasicComponentRegistryImpl$ComponentWrapper (org.infinispan.factories.impl)
preStart:254, GlobalComponentRegistry (org.infinispan.factories)
start:238, AbstractComponentRegistry (org.infinispan.factories)
internalStart:742, DefaultCacheManager (org.infinispan.manager)
start:713, DefaultCacheManager (org.infinispan.manager)
<init>:391, DefaultCacheManager (org.infinispan.manager)
<init>:335, DefaultCacheManager (org.infinispan.manager)
<init>:322, DefaultCacheManager (org.infinispan.manager)
init:64, IspnCacheManager (org.hawkular.alerts.cache)

And there the displayName is correct, while with Quarkus it isn't. I'm not sure what changed between 1.2.1.Final and 1.3.0.Final (Infinispan 10.0.0 -> 10.1.2.Final), but now the metrics are loaded multiple times and that breaks things.

Expected behavior
Infinispan would continue working as it did before.

Actual behavior
See above.

To Reproduce
Steps to reproduce the behavior:

1. https://github.com/burmanm/custom-policies-engine/tree/infinispan_changes
2. @Singleton's static { } blocks loads: cacheManager = new DefaultCacheManager(is);
3. Kaboom?

Configuration

Screenshots

Additional context

[geoand]

cc @jmartisk @karesti @wburns

[jmartisk]

This is the Infinispan commit after which it broke: https://github.com/infinispan/infinispan/commit/3dad7afef087aac2b2451678e8b78414c132af00#diff-06d9b67f22d98c28952c53d825f1d4f3R62-R74
Infinispan now starts registering base metrics by itself even though only Quarkus should be doing that. I think that Infinispan needs a new option to turn base metrics off.

[geoand]

@jmartisk what would be the effect of fixing #7907 in Quarkus with regards to this issue?

[jmartisk]

@geoand I think (didn't check) that then the metrics would just be registered twice, and the second registrations would overwrite the first ones, so while that would be unnecessary, in practice it would work correctly even without fixing the Infinispan part

[geoand]

That sounds a whole lot better than not being able to start the application at all :)

[jmartisk]

Hmm, so in the end, no it actually wouldn't work because there are Gauge metrics, and Gauges can't be re-registered without removing the original Gauge first... In that case I'm getting a different exception

Caused by: java.lang.IllegalArgumentException: A metric with metricID MetricID{name='classloader.loadedClasses.count', tags=[]} already exists
    at io.smallrye.metrics.MetricsRegistryImpl.register(MetricsRegistryImpl.java:131)
    at io.smallrye.metrics.MetricsRegistryImpl.register(MetricsRegistryImpl.java:111)
    at io.quarkus.smallrye.metrics.runtime.SmallRyeMetricsRecorder.classLoadingMetrics(SmallRyeMetricsRecorder.java:282)
    at io.quarkus.smallrye.metrics.runtime.SmallRyeMetricsRecorder.registerBaseMetrics(SmallRyeMetricsRecorder.java:111)
    at io.quarkus.deployment.steps.SmallRyeMetricsProcessor$registerBaseAndVendorMetrics7.deploy_0(SmallRyeMetricsProcessor$registerBaseAndVendorMetrics7.zig:65)
    at io.quarkus.deployment.steps.SmallRyeMetricsProcessor$registerBaseAndVendorMetrics7.deploy(SmallRyeMetricsProcessor$registerBaseAndVendorMetrics7.zig:36)
    at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:106)
    ... 13 more

[geoand]

OK, so we really need an Infinispan release to fix this issue I guess...

We'll also need some tests to make sure this doesn't happen again

[gsmet]

@jmartisk so we could unregister the gauges added by Infinispan to add our own? Are we sure ours will always be registered after the ones from Infinispan?

Asking that because I think we need a contingency plan in case we can't get an Infinispan release by the end of the month.

[gsmet]

Also maybe you should open an Infinispan issue describing the problem?

[gsmet]

And another option: if Infinispan now has its own metrics, maybe we should just remove the Quarkus one and let Infinispan do what they think is best for them?

[burmanm]

@gsmet That probably won't work, since Infinispan registers the GC, JVM, etc metrics. I don't think Quarkus users without Infinispan want to live without those metrics.

[gsmet]

Argh.

/cc @karesti @wburns
/cc @Sanne too while I'm at it

[anistor]

This is the Infinispan commit after which it broke: infinispan/infinispan@3dad7af#diff-06d9b67f22d98c28952c53d825f1d4f3R62-R74
Infinispan now starts registering base metrics by itself even though only Quarkus should be doing that. I think that Infinispan needs a new option to turn base metrics off.

This is something wrong that we noticed also and is already fixed in the newly released 10.1.5.Final

[anistor]

@gsmet That probably won't work, since Infinispan registers the GC, JVM, etc metrics. I don't think Quarkus users without Infinispan want to live without those metrics.

Infinispan embedded not longer does that in 10.1.5.final. Only the infinispan server continues to register those base metrics.

[wburns]

@gsmet I can try to get a PR with the updated version, but have to get my quarkus env working again :) Should have one soon hopefully, and assuming it passes tests.

[wburns]

I created https://github.com/quarkusio/quarkus/pull/7921 which fixes the client, but I just ran embedded and need to update some xml test files it appears.

[jmartisk]

I ran some experiments with #7921 included and I no longer can reproduce the problem. Perhaps @burmanm could you check it too and then we can close this issue?

[Sanne]

I'm confused about the premise:

the application that uses Infinispan directly (not through infinispan-embedded)

is this a supported configuration? Not as far as I know: one should either use the infinispan-embedded extension, or use the server. Incidentally, the infinispan-embedded extension was created by wburns exclusively to make Infinispan Server experiments with Quarkus.

Could you tell us more about your use case please?

[gsmet]

@Sanne I think he means he uses the Infinispan client extension and not the embedded one.

[Sanne]

That doesn't seem to be the case.

[anistor]

The ispn client does not expose MP metrics yet. That can't be the issue.

[burmanm]

@Sanne We create our own EmbeddedCacheManager in a different maven artifact than the one that runs Quarkus (with some logic which defines which configuration file is loaded).

This module is then embedded to our Quarkus service, which doesn't know about the Infinispan at all - what it sees is the interfaces to this module which may or may not use Infinispan (offtopic: and might indeed use a mixture of Infinispan and other services to manage the data, although I'm hoping to use Infinispan as a pretty RocksDB wrapper for most use-cases).

The other module has no knowledge of Quarkus and it's also used outside Quarkus. It makes a spaghetti code if I have to instantiate my Infinispan inside Quarkus only to start the other module by giving it the cacheManager (thus exposing internal implementation) and then do the same spaghetti code for every other runtime as well. That would pretty much be the end of any good OO principle.

[burmanm]

@jmartisk Yep, works with the Quarkus snapshot after #7921

[Sanne]

@burmanm you're bringing up some interesting points, we should probably write up some more about how we expect such designs to evolve; but I won't presume to write any general recommendations yet as many of these are novel ideas: for proper guidance to emerge we'll need more feedback like these and see what works best.

To put things into perspective: while this specific problem might have been solved already by #7921 , it's probably best you keep in mind that you're doing something we don't support and don't have integration tests for, so it's possible it will break again without notice. Let me try to explain why and how we expect designs to change - but again I won't presume we have thought of all aspects, so hopefully you'll be able to help figuring this all out.

In Quarkus it's fine to use 3rd party libraries which don't have an extension, but in this case:

• [obviously] we don't test for them
• we can't help you making them work in GraalVM native image
• you won't benefit from Live Reload capabilities, e.g. if the Infinispan configuration file is changed Quarkus won't reload
  
  
  
  • the specific component will not benefit from our optimisations, so possibly booting slower and consuming more runtime memory
  
  

For most "normal" Java code that people typically develop in house this is not a problem; an extension is only going to be needed for highly dynamic code, complex frameworks, libraries using bytecode enhancement, instrumentation, etc..

So while encapsulation is of course a highly valuable concept, I don't think we can compare instantiation of a complex service like an in-memory data grid cluster component with a straight-forward encapsulation need: such a component needs tight integration with the platform so its initialization, but also integration with management, metrics, operations need to "tie in" into the core platform. Typically one uses a service lookup for such aspects, and you're expected to have your own custom code "lookup" some service which is started and managed by the platform.

Wouldn't it be better to use the dependency injection mechanism? Quarkus offers great decoupling of services via CDI - in fact most of our own intregration is relying on it.

Offload the initialization and management details of such a component to the platform, and have your component simply inject a reference to the caches. Your other non-Quarkus services can easily provide a simimlar factory, thath's hardly figthing good OO principles.

Wouldn't that be better?