Spring-cloud-netflix: How to configure hostname verification in Zuul now?

[cah-oster]

I've been using Zuul with ribbon and SSL in version 1.0.6.RELEASE and tried upgrading to 1.1.0.RELEASE and run into hostname verification failures in my tests which use a localhost certificate for several hostnames. There was an earlier discussion on how to disable hostname verification here (https://github.com/spring-cloud/spring-cloud-netflix/pull/333#issuecomment-99164522) but those facilities are no longer present in 1.1.0.RELEASE. What is the current preferred way to configure the SSL settings for zuul's use of ribbon-discovered clients?

[ouaibsky]

Hi

on my project we switch all the yService platform to ssl and, haw to say it, it was not a funny game.

My advice would be to avoid disabling hostname certification because it's just pushing the sandbox a bit further.

We solve this by creating a shared keystore for dev environment, then we created a self-signed certificate for all dev hosts (with hostname aliases) then reading spring-boot documentation it is easiest to provide custom keystore, aliases, bla bla )
All this can be easily package int a basic bash script.

Regards
Christophe.

[cah-oster]

Thanks for the suggestion @ouaibsky. Yes I already am using a self-signed certificate and customizing the *stores, etc. I could probably create the additional host aliases for dev like this, but I'm also concerned I'm going to run into a similar (but slightly different) problem with our production configuration which uses consul service discovery. Service names are resolved into host ip:port rather than a hostname and won't match the certificate as the IPs are dynamic in our environment (docker-based).

[ouaibsky]

Your production concerns sounds different
If you configure your eureka client to register with dns hostname (i think it's done by default), you should not get any pb (and of course if your certificates is valid for this same dns name)
We deploied this way into an internal cloud and it works like a charm

Christophe

[cah-oster]

We aren't actually registering in consul with the spring stack, it's being done by mesos/marathon. I'll have to look if domain names can be provided. Thanks.

[pcornelissen]

When you are using the simpleHost rules you can have a look here:
https://github.com/orchit/zuulsslproblem/tree/18f71b2b789676229118b1612bf1f0f8b00bad6f

This is the commit to disable hostname verification in a simple zuul example.

[cah-oster]

@pcornelissen I tried a similar approach, but it didn't work for me. Did this work for you at runtime (beyond the simple mock test)?

I ran into a couple issues. The first was my bean for simpleHostRoutingFilter was being overridden/replaced by the one from ZuulProxyConfiguration, which I was able to overcome by extending and excluding it. The second was that it was still actually doing hostname verification, I believe because of this line in SimpleHostRoutingFilter (where the SSLConnectionSocketFactory constructor needs the hostnameverifier passed in):

        final Registry<ConnectionSocketFactory> registry = RegistryBuilder
                .<ConnectionSocketFactory> create()
                .register("http", PlainConnectionSocketFactory.INSTANCE)
                .register("https", new SSLConnectionSocketFactory(sslContext))
                .build();

[pcornelissen]

The ssl problem was gone but the while routing died :-(

[cah-oster]

@pcornelissen Ya, the SimpleHostRoutingFilter is not very extension-friendly. I ended up (for now), just copying the class into my zuul project, and editing it. My edits were 2 one-liners. First in the newClient() method, I added this to the builder:
.setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE)

Then in the newConnectionManager method, I replaced the registry https line like this:
.register("https", new SSLConnectionSocketFactory(sslContext,NoopHostnameVerifier.INSTANCE))

I missed the second change in my attempt to override and extend SimpleHostRoutingFilter, so perhaps that approach would work if you want to try. You can use an extension approach like in
https://github.com/spring-cloud/spring-cloud-netflix/blob/5d2065b3e764fcbbbba1d19994b983ea1c8de263/spring-cloud-netflix-core/src/test/java/org/springframework/cloud/netflix/zuul/filters/CustomHostRoutingFilterTests.java
Unfortunately it requires copy/paste large chunks of code to properly override those two methods, so I just went with the more explicit way for now of replacing the whole class. I don't know why #333 was closed without being incorporated; it seems to address this problem.

[pcornelissen]

I'll try to work on a PR for spring-cloud-netflix to resolve this properly hopefully this weekend

[bitsofinfo]

@pcornelissen @cah-oster

since 1.2.0 isn't out yet, and I need to get this to work now while using the released 1.1.13/14 line, I am trying what you did above (copy the class into my local project and add the support there).

• Copied the patched SimpleHostRoutingFilter to my local project (i.e. same class/package name)
• Modified it to manually get the zuul.sslHostnameValidationEnabled system property
• Verified in a debugger that this block is indeed running:

....
else {
    registryBuilder.register("https", new SSLConnectionSocketFactory(
    sslContext, NoopHostnameVerifier.INSTANCE));
}

Doing so still does not seem to work for me, I constantly am getting errors like the below.

When I step in the SocketFactory in this thread, is still using the BrowserCompatHostnameVerifier

Daemon Thread [DiscoveryClient-CacheRefreshExecutor-0] (Suspended (breakpoint at line 177 in DefaultClientConnectionOperator))  
    DefaultClientConnectionOperator.openConnection(OperatedClientConnection, HttpHost, InetAddress, HttpContext, HttpParams) line: 177  
    BasicPoolEntry(AbstractPoolEntry).open(HttpRoute, HttpContext, HttpParams) line: 144    
    BasicPooledConnAdapter(AbstractPooledConnAdapter).open(HttpRoute, HttpContext, HttpParams) line: 131    
    DefaultRequestDirector.tryConnect(RoutedRequest, HttpContext) line: 611 
    DefaultRequestDirector.execute(HttpHost, HttpRequest, HttpContext) line: 446    
    DefaultHttpClient(AbstractHttpClient).doExecute(HttpHost, HttpRequest, HttpContext) line: 882   
    DefaultHttpClient(CloseableHttpClient).execute(HttpHost, HttpRequest) line: 117 
    DefaultHttpClient(CloseableHttpClient).execute(HttpHost, HttpRequest) line: 55  
    ApacheHttpClient4Handler.handle(ClientRequest) line: 173    
    GZIPContentEncodingFilter.handle(ClientRequest) line: 123   
    EurekaIdentityHeaderFilter.handle(ClientRequest) line: 27   
    ApacheHttpClient4(Client).handle(ClientRequest) line: 652   
    WebResource.handle(Class<T>, ClientRequest) line: 682   
    WebResource.access$200(WebResource, Class, ClientRequest) line: 74  
    WebResource$Builder.get(Class<T>) line: 509 
    JerseyApplicationClient(AbstractJerseyEurekaHttpClient).getApplicationsInternal(String, String[]) line: 194 
    JerseyApplicationClient(AbstractJerseyEurekaHttpClient).getDelta(String...) line: 170   
    EurekaHttpClientDecorator$7.execute(EurekaHttpClient) line: 152 
    MetricsCollectingEurekaHttpClient.execute(RequestExecutor<R>) line: 73  
    MetricsCollectingEurekaHttpClient(EurekaHttpClientDecorator).getDelta(String...) line: 149  
    EurekaHttpClientDecorator$7.execute(EurekaHttpClient) line: 152 
    RedirectingEurekaHttpClient.execute(RequestExecutor<R>) line: 89    
    RedirectingEurekaHttpClient(EurekaHttpClientDecorator).getDelta(String...) line: 149    
    EurekaHttpClientDecorator$7.execute(EurekaHttpClient) line: 152 
    RetryableEurekaHttpClient.execute(RequestExecutor<R>) line: 119 
    RetryableEurekaHttpClient(EurekaHttpClientDecorator).getDelta(String...) line: 149  
    EurekaHttpClientDecorator$7.execute(EurekaHttpClient) line: 152 
    SessionedEurekaHttpClient.execute(RequestExecutor<R>) line: 77  
    SessionedEurekaHttpClient(EurekaHttpClientDecorator).getDelta(String...) line: 149  
    CloudEurekaClient(DiscoveryClient).getAndUpdateDelta(Applications) line: 1054   
    CloudEurekaClient(DiscoveryClient).fetchRegistry(boolean) line: 936 
    CloudEurekaClient(DiscoveryClient).refreshRegistry() line: 1456 
    DiscoveryClient$CacheRefreshThread.run() line: 1423 
    Executors$RunnableAdapter<T>.call() line: 511   
    FutureTask<V>.run() line: 266   
    ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1142  
    ThreadPoolExecutor$Worker.run() line: 617   
    Thread.run() line: 745  

Caused by: com.sun.jersey.api.client.ClientHandlerException: javax.net.ssl.SSLException: Certificate for <192.168.1.148> doesn't match common name of the certificate subject: my-test-cert
    at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:187)
    at com.sun.jersey.api.client.Client.handle(Client.java:652)
    at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
    at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
    at com.sun.jersey.api.client.WebResource$Builder.put(WebResource.java:539)
    at com.netflix.niws.client.http.RestClient.execute(RestClient.java:624)
    at com.netflix.niws.client.http.RestClient.execute(RestClient.java:527)
    at com.netflix.niws.client.http.RestClient.execute(RestClient.java:92)
    at com.netflix.client.AbstractLoadBalancerAwareClient$1.call(AbstractLoadBalancerAwareClient.java:109)
    at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:303)
    at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:287)
    at rx.internal.util.ScalarSynchronousObservable$4.call(ScalarSynchronousObservable.java:223)
    at rx.internal.util.ScalarSynchronousObservable$4.call(ScalarSynchronousObservable.java:220)
    at rx.Observable.unsafeSubscribe(Observable.java:8460)
    at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.drain(OnSubscribeConcatMap.java:286)
    at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.onNext(OnSubscribeConcatMap.java:144)
    at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:185)
    at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180)
    at rx.Observable.unsafeSubscribe(Observable.java:8460)
    at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94)
    at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42)
    at rx.Observable.unsafeSubscribe(Observable.java:8460)
    at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber$1.call(OperatorRetryWithPredicate.java:131)
    at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:76)
    at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:55)
    at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:83)
    at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:49)
    at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:268)
    at rx.Subscriber.setProducer(Subscriber.java:209)
    at rx.internal.util.ScalarSynchronousObservable$1.call(ScalarSynchronousObservable.java:79)
    at rx.internal.util.ScalarSynchronousObservable$1.call(ScalarSynchronousObservable.java:75)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.subscribe(Observable.java:8553)
    at rx.Observable.subscribe(Observable.java:8520)
    at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:433)
    at rx.observables.BlockingObservable.single(BlockingObservable.java:332)
    at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:102)
    ... 157 common frames omitted
Caused by: javax.net.ssl.SSLException: Certificate for <192.168.1.148> doesn't match common name of the certificate subject: kernel-ssl-test
    at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:172)
    at org.apache.http.conn.ssl.BrowserCompatHostnameVerifier.verify(BrowserCompatHostnameVerifier.java:61)
    at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:140)
    at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:114)
    at org.apache.http.conn.ssl.SSLSocketFactory.verifyHostname(SSLSocketFactory.java:569)
    at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:544)
    at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
    at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
    at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
    at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
    at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:117)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:173)
    ... 198 common frames omitted

[cah-oster]

@bitsofinfo This fix was actually in 1.1.3, and rolled up into spring-cloud Brixton.SR2. I was able to remove my hack, move to Brixton.SR2, just set this in my application.properties:

zuul:
  sslHostnameValidationEnabled: false

1.1.4 and Brixton.SR3 are now also available which should work as well.

[bitsofinfo]

hmm, Im running with SR3 and still getting the above kind of error with

-Dzuul.sslHostnameValidationEnabled=false

Note the thread stack above, this is originating from a thread invoking DiscoveryClient not sure if that is the same thread path that your original fix was intended to address

Also @cah-oster any thought to allowing someone to provide a _custom_ HostnameVerifier rather than the all or nothing with the NoOp one? I have a case where I _only_ want to provide an exception for one known CN on a specific cert, all the rest can be deferred to the normal rules in apache's default BrowserCompatHostnameVerifier, but this solution is just all or nothing.

[spencergibb]

sslHostnameValidationEnabled only works for SimpleHostRoutingFilter not the ribbon routing filter.

[bitsofinfo]

k, so perhaps its related to https://github.com/Netflix/eureka/issues/812 ?

[jmgilmour]

Apologies for posting on an old issue (and I can create a new issue if necessary), but @spencergibb how would I configure hostname validation on the ribbon routing filter?

[jmgilmour]

I found an example in the comments of https://github.com/spring-cloud/spring-cloud-netflix/issues/1776

[kingsant]

spring-cloud-netflix-core:1.1.0.RELEASE
In this version,sslHostnameValidationEnabled donot wrok!
which version do it work?
if i can overwrite hostnameverify to avoid this proplem?
@spencergibb @pcornelissen pcornelissen

[zrss]

@bitsofinfo This fix was actually in 1.1.3, and rolled up into spring-cloud Brixton.SR2. I was able to remove my hack, move to Brixton.SR2, just set this in my application.properties:

zuul:
  sslHostnameValidationEnabled: false

1.1.4 and Brixton.SR3 are now also available which should work as well.

@kingsant