Kubernetes-client: KubernetesClient inside of Pod

[rainer-maierhofer]

How can I create a KubernetesClient that is running inside of a pod. Is there a way to automatically get the configuration settings?

[sulaukad]

+1
Also, looking for some sort of faq about this

[guilhermecgs]

+1

[manusa]

We're planning on providing an example and documentation for our current Sprint

[rohanKanojia]

@guilhermecgs @sulaukad @rainer-maierhofer : Hi guys, Do you know if this is possible in go-client? I'm not sure how we would be reading config inside Kubernetes. If it is possible, Could you please share example?

[rainer-maierhofer]

Hello Rohan, I used io.kubernetes java client instead.
It has a default client that you can use inside a pod. This works fine for me, even if the api is not as good.

[manusa]

Just for reference:
https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/#directly-accessing-the-rest-api-1

[rohanKanojia]

@rainer-maierhofer : ah, you moved to official client :broken_heart: . Let me try this out whenever I get time. Need to check how to replicate these:
https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod
https://github.com/kubernetes/client-go/blob/master/examples/in-cluster-client-configuration/main.go

[guilhermecgs]

@manusa , my use case is using this java kubernetes-client inside a pod to perform some GET operations in the API.

Somehow it worked out of the box without having to use any password whatsoever, but now I need more detail and fine adjustments.. For example, to use a different service account and others config/security settings.

[rainer-maierhofer]

DefaultKubernetesClient uses the default value for kubeconfig which is ~/.kube/config. In production this value is not set.

[guilhermecgs]

In production this value is not set.

By definition, the ~/.kube/config folder is not set in production?

I received this error in production environment

i.f.k.c.KubernetesClientException: Failure executing: GET at: https://XX.XX.0.1/api/v1/services?labelSelector=XXXX-XXX%3Dtrue. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. services is forbidden: User "system:serviceaccount:prd-YYY-YYY:default" cannot list services at the cluster scope: User "system:serviceaccount:prd-YYY-YYY:default" cannot list all services in the cluster.

[sulaukad]

Hi guys, I've stuck with the handshake issue while trying to create a secret from inside and get such
exception

Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) ~[na:na]
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) ~[na:na]
at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) ~[na:na]
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Unknown Source) ~[na:na]
at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source) ~[na:na]
at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) ~[na:na]
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:336) ~[okhttp-3.14.7.jar:na]
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300) ~[okhttp-3.14.7.jar:na]
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185) ~[okhttp-3.14.7.jar:na]
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224) ~[okhttp-3.14.7.jar:na]
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108) ~[okhttp-3.14.7.jar:na]
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88) ~[okhttp-3.14.7.jar:na]
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169) ~[okhttp-3.14.7.jar:na]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41) ~[okhttp-3.14.7.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.7.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.7.jar:na]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[okhttp-3.14.7.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.7.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.7.jar:na]

I'm using default configuration with the secret configuration from .yaml using

client.load(secret)
.inNamespace(client.getNamespace())
.createOrReplace();

Any ideas about this?
Thanks!

[rainer-maierhofer]

Have a look at this:
https://stackoverflow.com/questions/2145431/https-using-jersey-client

[sulaukad]

@rainer-maierhofer
Is there any way to use default okhttp configuration, with some env variables provided?

[rohanKanojia]

Hi guys, I tried out doing this and was able to use Fabric8 Kubernetes Client from within a Pod. I have created a simple project for this : https://github.com/rohanKanojia/kubernetes-client-inside-pod

Initially I also was facing this error:

  | "Failure executing: GET at: https://10.96.0.1/api/v1/namespaces/default/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"."
-- | --

Only additional step I needed to do was to give Pod's ServiceAccount correct privileges:

kubectl create clusterrolebinding default-pod --clusterrole cluster-admin --serviceaccount=default:default

I was able to use it afterwards.

[rainer-maierhofer]

@rohanKanojia
Your sample uses the default value for kubeconfig which is ~/.kube/config. In production this value is not set.
Is there a way to configure the kubeconfig automaticaly similar to io.kubernetes.

[rohanKanojia]

You can override default value by using property named KUBECONFIG. See https://github.com/fabric8io/kubernetes-client#configuring-the-client

[rainer-maierhofer]

io.kubernetes java client offers a default configuration inside a pod that I can use, even if I do not know the kubernetes master url. How can I find out the correct config values, when I am not the administrator?
It would be nice to have similar solution in fabric8io.

[rohanKanojia]

Before looking into this, I checked out official client's InClusterConfig method. It just makes up a configuration object like this: https://github.com/kubernetes/client-go/blob/master/rest/config.go#L451

Initially, I thought I just need to add another method in Config class similar to this. But when I looked at Config class, it was already handling all these things, default URL is mapped to KUBERNETES_SERVICE_HOST environment variable[0]. Pod's Service account is loaded here[1]

[0] https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java#L401
[1] https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java#L113

[rohanKanojia]

So I would say it loads with ServiceAccount which is assigned to your Pod. It all depends on whether your ServiceAccount has access to do something or not. I think this is when cluster administrator comes in picture. You would need to configure your Pod to use some other ServiceAccount which has access rather than default one.

        Pod podWithSomeOtherSa = new PodBuilder()
                .withNewMetadata().withName("nginx").endMetadata()
                .withNewSpec()
                .addNewContainer()
                .withImage("nginx")
                .withName("nginx")
                .endContainer()
                .withServiceAccountName("build-robot") // Service Account which might be created by cluster admin for these kind of tasks
                .endSpec()
                .build();

It mounts ServiceAccount related details as a volume in /var/run/secrets/kubernetes.io/serviceaccount/

[sulaukad]

@rohanKanojia @rainer-maierhofer
FYI
About handshake issue
Kubernetes API was missing EC ciphers which do not ship with Correto distribution
I've added a bouncycastle dependency to my build.gradle registered it in java

implementation "org.bouncycastle:bcprov-jdk15on"

Security.insertProviderAt(new BouncyCastleProvider(), 1)

Also, the client works without any issues from inside with the default configuration from examples. It uses token and cert from /var/run/secrets/kubernetes.io/serviceaccount/ . You just need to setup RBAC for your deployment