Spring-boot: Configuring dedicated Cassandra contact points now require a local DC

[snicoll]

We need to expose an additional property as the following is thrown when one wants to customize the contact points to use.

Caused by: java.lang.IllegalStateException: Since you provided explicit contact points, the local DC must be explicitly set (see basic.load-balancing-policy.local-datacenter in the config, or set it programmatically with SessionBuilder.withLocalDatacenter). Current contact points are: Node(endPoint=/127.0.0.1:9042, hostId=3e9d107c-8fab-4458-b217-d84c4491b574, hashCode=22d286ac)=datacenter1. Current DCs in this cluster are: datacenter1
    at com.datastax.oss.driver.internal.core.loadbalancing.helper.MandatoryLocalDcHelper.discoverLocalDc(MandatoryLocalDcHelper.java:91) ~[java-driver-core-4.3.1.jar:na]
    at com.datastax.oss.driver.internal.core.loadbalancing.DefaultLoadBalancingPolicy.discoverLocalDc(DefaultLoadBalancingPolicy.java:73) ~[java-driver-core-4.3.1.jar:na]
    at com.datastax.oss.driver.internal.core.loadbalancing.BasicLoadBalancingPolicy.init(BasicLoadBalancingPolicy.java:126) ~[java-driver-core-4.3.1.jar:na]
    at com.datastax.oss.driver.internal.core.metadata.LoadBalancingPolicyWrapper.init(LoadBalancingPolicyWrapper.java:121) ~[java-driver-core-4.3.1.jar:na]
    at com.datastax.oss.driver.internal.core.session.DefaultSession$SingleThreaded.afterInitialSchemaRefresh(DefaultSession.java:419) ~[java-driver-core-4.3.1.jar:na]
    at com.datastax.oss.driver.internal.core.session.DefaultSession$SingleThreaded.lambda$afterInitialNodeListRefresh$5(DefaultSession.java:409) ~[java-driver-core-4.3.1.jar:na]
    at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[na:1.8.0_232]
    at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646) ~[na:1.8.0_232]
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[na:1.8.0_232]
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[na:1.8.0_232]
    at com.datastax.oss.driver.internal.core.metadata.MetadataManager$SingleThreaded.lambda$startSchemaRequest$2(MetadataManager.java:420) ~[java-driver-core-4.3.1.jar:na]
    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[na:1.8.0_232]
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[na:1.8.0_232]
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[na:1.8.0_232]
    at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575) ~[na:1.8.0_232]
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:594) ~[na:1.8.0_232]
    at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456) ~[na:1.8.0_232]
    at io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54) ~[netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) ~[netty-common-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.43.Final.jar:4.1.43.Final]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_232]

At the moment this even happens by default (i.e. without changing anything) so that's quite annoying.

[mp911de]

@adutra can you have a look?

[snicoll]

If I specify a local DC as instructed via a customizer, I get the following warning:

2020-01-16 14:11:43.970  WARN 30497 --- [     s0-admin-0] c.d.o.d.i.c.l.h.OptionalLocalDcHelper    : [s0|default] You specified test as the local DC, but some contact points are from a different DC: Node(endPoint=/127.0.0.1:9042, hostId=3e9d107c-8fab-4458-b217-d84c4491b574, hashCode=27bd52ff)=datacenter1; please provide the correct local DC, or check your contact points

This is running with only the defaults targeting cassandra running on my machine.

[snicoll]

If you specify a contact-point with a port, it seems it is ignored.

spring.data.cassandra.contact-points=localhost:4534

leads to

Caused by: com.datastax.oss.driver.api.core.AllNodesFailedException: Could not reach any contact point, make sure you've provided valid addresses (showing first 2, use getErrors() for more: Node(endPoint=localhost/0:0:0:0:0:0:0:1:4534, hostId=null, hashCode=7a1757e7): com.datastax.oss.driver.api.core.connection.ConnectionInitException: [s0|control|connecting...] init query OPTIONS: error writing , Node(endPoint=localhost/127.0.0.1:4534, hostId=null, hashCode=7929ae7a): com.datastax.oss.driver.api.core.connection.ConnectionInitException: [s0|control|connecting...] init query OPTIONS: error writing )
    at com.datastax.oss.driver.api.core.AllNodesFailedException.copy(AllNodesFailedException.java:95) ~[java-driver-core-4.3.1.jar:na]
    at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149) ~[java-driver-core-4.3.1.jar:na]
    at com.datastax.oss.driver.api.core.session.SessionBuilder.build(SessionBuilder.java:501) ~[java-driver-core-4.3.1.jar:na]
    at org.springframework.boot.autoconfigure.cassandra.CassandraAutoConfiguration.cassandraSession(CassandraAutoConfiguration.java:67) ~[spring-boot-autoconfigure-2.3.0.BUILD-20200116.094724-144.jar:2.3.0.BUILD-SNAPSHOT]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_232]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_232]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_232]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_232]
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    ... 33 common frames omitted

However, the following

spring.data.cassandra.contact-points=something

starts the app just fine (as I happen to have cassandra running on the default port):

2020-01-16 14:16:56.216  INFO 32270 --- [           main] c.d.o.d.i.core.DefaultMavenCoordinates   : DataStax Java driver for Apache Cassandra(R) (com.datastax.oss:java-driver-core) version 4.3.1
2020-01-16 14:16:56.600  INFO 32270 --- [     s0-admin-0] c.d.oss.driver.internal.core.time.Clock  : Using native clock for microsecond precision
2020-01-16 14:16:56.603  INFO 32270 --- [     s0-admin-0] c.d.o.d.i.core.metadata.MetadataManager  : [s0] No contact points provided, defaulting to /127.0.0.1:9042

[adutra]

All DataStax drivers decided in the past years to move away from 1) built-in DC failover and 2) automatic local DC inference from contact points. In the Java driver, these decisions were applied starting with driver 4.0. This is because we firmly believe that both features can lead to misconfiguration issues.

The driver now requires from the user to explicitly specify the local DC. You can do so either through configuration with the option basic.load-balancing-policy.local-datacenter or programmatically, with SessionBuilder.withLocalDataCenter. Check our docs on load balancing for more information.

I would recommend that you also require from Spring users to explicitly define the local datacenter. If that’s not an option for you, the workaround would be to use an alternate load balancing policy that ships with the driver as well: DcInferringLoadBalancingPolicy. As its name implies, this LBP would assume that the local DC is whatever DC the contact points report. But this LBP is not meant as a general-purpose policy, it’s rather for analytics tools (Spark, mostly).

Requiring a local DC may pose problems for some of your tests because embedded Cassandra libraries tend to assign arbitrary datacenter names to test clusters. If that’s the case, the simplest solution is to actually _not_ specify any contact point, in which case the default LBP will switch to “test mode” (so to speak) and assume that 1) the contact point is localhost and 2) the local DC is whatever DC that contact point reports.

[adutra]

If I specify a local DC as instructed via a customizer, I get the following warning:

2020-01-16 14:11:43.970  WARN 30497 --- [     s0-admin-0] c.d.o.d.i.c.l.h.OptionalLocalDcHelper    : [s0|default] You specified test as the local DC, but some contact points are from a different DC: Node(endPoint=/127.0.0.1:9042, hostId=3e9d107c-8fab-4458-b217-d84c4491b574, hashCode=27bd52ff)=datacenter1; please provide the correct local DC, or check your contact points

This is running with only the defaults targeting cassandra running on my machine.

The correct datacenter seems to be datacenter1 in your case, but you specified test instead. These misconfigurations are very common in real life, and the main reason why we decided to be more opinionated about datacenter names. Connecting to the wrong datacenter may be extremely impactful on performance.

[adutra]

If you specify a contact-point with a port, it seems it is ignored.

init query OPTIONS: error writing most certainly means that the socket could not be opened, or an SSL handshake error happened. We improved the error messages in driver 4.4.0. In any case, the port is probably wrong.

starts the app just fine (as I happen to have cassandra running on the default port):

Do you mind double checking? It seems to me that the contact point wasn’t passed at all to the driver. This is why it reports “no contact points provided” and switches to “test mode”, which in turn makes the connection successful.

[jmooo]

If anyone comes across this ticket googling the spring 2.3 local DC error, the spring config var you want is:

spring.data.cassandra.local-datacenter=datacenter1

(where datacenter1 is whatever you previously named your DC)

[m-kajiwara]

When I specified multiple contact points, setLocalDatacenter set only the first one?
When I used DefaultDriverOption.LOAD_BALANCING_LOCAL_DATACENTER like CassandraAutoConfiguration, it's ok
Or addContactPoints(List).withLocalDatacenter()

    String getContactPoints() {
        return "cassandra-node1,cassandra-node2,cassandra-node3"
    }
    String getLocalDataCenter() {
        return "dc1";
    }

You specified dc1 as the local DC, but some contact points are from a different DC: Node(endPoint=cassandra-node2:9042, hostId=null, hashCode=)=null, Node(endPoint=cassandra-node3:9042, hostId=null, hashCode=)=null;

[snicoll]

I think this is a question for the Cassandra support as things have changed quite a lot with the new driver. If they can’t help and you believe you’ve found an issue, please open a separate issue.

[adutra]

When I specified multiple contact points, setLocalDatacenter set only the first one?

The local datacenter must be the same for all contact points.

It's hard to tell without more context, but it does look like you are hitting JAVA-2835. It will be fixed in the next driver release, 4.8.0, due in the next few days.