Orientdb: OrientDB w/hazelcast is unable to start on a docker container due to the public-address being the address of the host not the container

[gryffon]

OrientDB Version: 2.2.36

Java Version: OpenJDK Runtime Environment (build 1.8.0_191-b12)

OS: CentOS Linux release 7.6.1810 (Core)

Expected behavior

Have someway to support NAT-like behavior in OrientDB w/hazelcast and docker.

Actual behavior

When supplying a via the hazelcast.xml, OrientDB uses this to try and bind it's listener. In this case, the public-address is external to the container, and this causes a failure.

OrientDB in this case, is being run as part of Sonatype Nexus 3. The docker container is not in any kind of swarm configuration, it is intended to connect to other hosts that will have the same setup.

Log of output:
2019-03-27 19:33:18,235+0000 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl - OrientDB version: 2.2.36 (build d3beb772c02098ceaea89779a7afd4b7305d3788, branch 2.2.x)
2019-03-27 19:33:18,941+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.instance.AddressPicker - [LOCAL] [nexus] [3.10.3] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [10.250.XX.XX, 10.248.XX.XX, 10.250.XX.XX]
2019-03-27 19:33:18,941+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.instance.AddressPicker - [LOCAL] [nexus] [3.10.3] Prefer IPv4 stack is true.
2019-03-27 19:33:18,943+0000 WARN [FelixStartLevel] *SYSTEM com.hazelcast.instance.AddressPicker - [LOCAL] [nexus] [3.10.3] Could not find a matching address to start with! Picking one of non-loopback addresses.
2019-03-27 19:33:18,952+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.instance.AddressPicker - [LOCAL] [nexus] [3.10.3] Picked [172.25.0.3]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true
2019-03-27 19:33:18,953+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.instance.AddressPicker - [LOCAL] [nexus] [3.10.3] Using public address: [10.248.XX.XX]:5701
2019-03-27 19:33:18,966+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.system - [10.248.XX.XX]:5701 [nexus] [3.10.3] Hazelcast 3.10.3 (20180718 - fec4eef) starting at [10.248.XX.XX]:5701
2019-03-27 19:33:18,966+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.system - [10.248.XX.XX]:5701 [nexus] [3.10.3] Copyright (c) 2008-2018, Hazelcast, Inc. All Rights Reserved.
2019-03-27 19:33:18,966+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.system - [10.248.XX.XX]:5701 [nexus] [3.10.3] Configured Hazelcast Serialization version: 1
2019-03-27 19:33:18,966+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.instance.Node - [10.248.XX.XX]:5701 [nexus] [3.10.3] A non-empty group password is configured for the Hazelcast member. Starting with Hazelcast version 3.8.2, members with the same group name, but with different group passwords (that do not use authentication) form a cluster. The group password configuration will be removed completely in a future release.
2019-03-27 19:33:19,100+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator - [10.248.XX.XX]:5701 [nexus] [3.10.3] Backpressure is disabled
2019-03-27 19:33:19,114+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.spi.impl.operationservice.impl.InboundResponseHandlerSupplier - [10.248.XX.XX]:5701 [nexus] [3.10.3] Running with 2 response threads
2019-03-27 19:33:19,439+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.instance.Node - [10.248.XX.XX]:5701 [nexus] [3.10.3] Creating TcpIpJoiner
2019-03-27 19:33:19,568+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl - [10.248.XX.XX]:5701 [nexus] [3.10.3] Starting 1 partition threads and 2 generic threads (1 dedicated for priority tasks)
2019-03-27 19:33:19,572+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.internal.diagnostics.Diagnostics - [10.248.XX.XX]:5701 [nexus] [3.10.3] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2019-03-27 19:33:19,576+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.core.LifecycleService - [10.248.XX.XX]:5701 [nexus] [3.10.3] [10.248.XX.XX]:5701 is STARTING
2019-03-27 19:33:19,621+0000 INFO [hz.nexus.cached.thread-3] *SYSTEM com.hazelcast.nio.tcp.TcpIpConnector - [10.248.XX.XX]:5701 [nexus] [3.10.3] Connecting to /10.250.XX.XX:5701, timeout: 0, bind-any: true
2019-03-27 19:33:19,621+0000 INFO [hz.nexus.cached.thread-2] *SYSTEM com.hazelcast.nio.tcp.TcpIpConnector - [10.248.XX.XX]:5701 [nexus] [3.10.3] Connecting to /10.250.XX.XX:5701, timeout: 0, bind-any: true
2019-03-27 19:33:19,701+0000 INFO [hz.nexus.cached.thread-3] *SYSTEM com.hazelcast.nio.tcp.TcpIpConnector - [10.248.XX.XX]:5701 [nexus] [3.10.3] Could not connect to: /10.250.XX.XX:5701. Reason: SocketException[Connection refused to address /10.250.XX.XX:5701]
2019-03-27 19:33:19,701+0000 INFO [hz.nexus.cached.thread-3] *SYSTEM com.hazelcast.cluster.impl.TcpIpJoiner - [10.248.XX.XX]:5701 [nexus] [3.10.3] [10.250.XX.XX]:5701 is added to the blacklist.
2019-03-27 19:33:24,615+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.system - [10.248.XX.XX]:5701 [nexus] [3.10.3] Cluster version set to 3.10
2019-03-27 19:33:24,616+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.internal.cluster.ClusterService - [10.248.XX.XX]:5701 [nexus] [3.10.3]

Members {size:1, ver:1} [
Member [10.248.XX.XX]:5701 - 9f6456b6-6a73-4552-ab0c-81148a7d7aaa this
]

2019-03-27 19:33:24,647+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.internal.management.ManagementCenterService - [10.248.XX.XX]:5701 [nexus] [3.10.3] Hazelcast will connect to Hazelcast Management Center on address:
http://localhost:8080/
2019-03-27 19:33:24,649+0000 INFO [hz.nexus.MC.Task.Poller] *SYSTEM com.hazelcast.internal.management.ManagementCenterService - [10.248.XX.XX]:5701 [nexus] [3.10.3] Failed to pull tasks from management center
2019-03-27 19:33:24,650+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.internal.jmx.ManagementService - [10.248.XX.XX]:5701 [nexus] [3.10.3] Hazelcast JMX agent enabled.
2019-03-27 19:33:24,669+0000 INFO [FelixStartLevel] *SYSTEM com.hazelcast.core.LifecycleService - [10.248.XX.XX]:5701 [nexus] [3.10.3] [10.248.XX.XX]:5701 is STARTED
2019-03-27 19:33:24,669+0000 INFO [FelixStartLevel] *SYSTEM com.sonatype.nexus.hazelcast.internal.HazelcastInstanceProvider - Instance: HazelcastInstance{name='nexus', node=[10.248.XX.XX]:5701}
2019-03-27 19:33:24,684+0000 INFO [hz.nexus.MC.State.Sender] *SYSTEM com.hazelcast.internal.management.ManagementCenterService - [10.248.XX.XX]:5701 [nexus] [3.10.3] Failed to connect to:http://localhost:8080/collector.do
2019-03-27 19:33:24,685+0000 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl - Binary listener enabled: 10.248.XX.XX:[2424-2430]
2019-03-27 19:33:24,688+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - OrientDB Server v2.2.36 (build d3beb772c02098ceaea89779a7afd4b7305d3788, branch 2.2.x) is starting up...
2019-03-27 19:33:24,695+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - Databases directory: /nexus-data/db
2019-03-27 19:33:24,935+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.engine.OMemoryAndLocalPaginatedEnginesInitializer - Configuration of usage of soft references inside of containers of results of SQL execution
2019-03-27 19:33:24,936+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.engine.OMemoryAndLocalPaginatedEnginesInitializer - Initial and maximum values of heap memory usage are equal, containers of results of SQL executors will use soft references by default
2019-03-27 19:33:24,936+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.engine.OMemoryAndLocalPaginatedEnginesInitializer - Auto configuration of disk cache size.
2019-03-27 19:33:25,006+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.common.jna.ONative - 8182038528 B/7803 MB/7 GB of physical memory were detected on machine
2019-03-27 19:33:25,020+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.common.jna.ONative - Soft memory limit for this process is set to -1 B/-1 MB/-1 GB
2019-03-27 19:33:25,022+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.common.jna.ONative - Hard memory limit for this process is set to -1 B/-1 MB/-1 GB
2019-03-27 19:33:25,022+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.common.jna.ONative - Path to 'memory' cgroup is '/docker/fbc4a0ec0c2be50a3f1a4b4babf7802851ff248ebbd21029d907c99d08877e2f'
2019-03-27 19:33:25,023+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.common.jna.ONative - Mounting path for memory cgroup controller is '/sys/fs/cgroup/memory'
2019-03-27 19:33:25,024+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.common.jna.ONative - Can not find '/sys/fs/cgroup/memory/docker/fbc4a0ec0c2be50a3f1a4b4babf7802851ff248ebbd21029d907c99d08877e2f' path for memory cgroup, it is supposed that process is running in container, will try to read root '/sys/fs/cgroup/memory' memory cgroup data
2019-03-27 19:33:25,024+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.common.jna.ONative - cgroup soft memory limit is 9223372036854771712 B/8796093022207 MB/8589934591 GB
2019-03-27 19:33:25,024+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.common.jna.ONative - cgroup hard memory limit is 9223372036854771712 B/8796093022207 MB/8589934591 GB
2019-03-27 19:33:25,024+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.common.jna.ONative - Detected memory limit for current process is 8182038528 B/7803 MB/7 GB
2019-03-27 19:33:25,029+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.engine.OMemoryAndLocalPaginatedEnginesInitializer - OrientDB auto-config DISKCACHE=2,048MB (heap=1,094MB direct=2,048MB os=7,803MB)
2019-03-27 19:33:25,033+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.config.OGlobalConfiguration - Lowering disk cache size from 2,048MB to 2,046MB.
2019-03-27 19:33:25,189+0000 WARN [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.network.OServerNetworkListener - Port 10.248.XX.XX:2424 busy, trying the next available...
2019-03-27 19:33:25,190+0000 WARN [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.network.OServerNetworkListener - Port 10.248.XX.XX:2425 busy, trying the next available...
2019-03-27 19:33:25,190+0000 WARN [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.network.OServerNetworkListener - Port 10.248.XX.XX:2426 busy, trying the next available...
2019-03-27 19:33:25,192+0000 WARN [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.network.OServerNetworkListener - Port 10.248.XX.XX:2427 busy, trying the next available...
2019-03-27 19:33:25,192+0000 WARN [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.network.OServerNetworkListener - Port 10.248.XX.XX:2428 busy, trying the next available...
2019-03-27 19:33:25,192+0000 WARN [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.network.OServerNetworkListener - Port 10.248.XX.XX:2429 busy, trying the next available...
2019-03-27 19:33:25,192+0000 WARN [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.network.OServerNetworkListener - Port 10.248.XX.XX:2430 busy, trying the next available...
2019-03-27 19:33:25,192+0000 ERROR [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.network.OServerNetworkListener - Unable to listen for connections using the configured ports '2424-2430' on host '10.248.XX.XX'

Steps to reproduce

Run OrientDB w/hazelcast inside a docker container and set the public-address to the external ip address of the host(not the container).

[gtonic]

+1

[Tangresh]

This is a problem with how Sonatype Nexus Pro configures its OrientDB instance. The responsible code is part of the (proprietary) nexus-hazelcast-plugin.
The corresponding Sonatype Issue tracker ticket is closed with a resolution of "Won't close": https://issues.sonatype.org/browse/NEXUS-20937