Ohara: OharaStream 0.6.0 workspace無法建立

[louie5486]

我的環境-
OS:centos 7.6
kuberos env: v1.14.1
docker env:v19.03

nodes: 1 master + 3 works
NAME STATUS ROLES AGE VERSION
ohara.com.tw Ready master 107m v1.14.1
ohwork1.com.tw Ready 101m v1.14.1
ohwork2.com.tw Ready 97m v1.14.1
ohwork3.com.tw Ready 94m v1.14.1

oharastream images:
oharastream/configurator:0.6.0
oharastream/manager:0.6.0
oharastream/zookeeper:0.6.0
oharastream/broker:0.6.0
docker pull oharastream/connect-worker:0.6.0
docker pull oharastream/streamapp:0.6.0

問題:
依照github說明文件，準備好k8s環境，並且啟用oharastream configrator & manager後，
完成web介面上node新增作業，後續要進行workspace新增作業時，作業失敗並且出現一些error message。

操作如下：

configrator error log:

2019-08-12 08:55:11,370 INFO [pool-1-thread-4] k8s.K8SClient$$anon$1$$anon$2(303): create pod request json: {"apiVersion":"v1","kind":"Pod","metadata":{"name":"k8soccl-gkmdruxos0-zk-2dee0d8","labels":{"name":"ohara"}},"spec":{"hostname":"k8soccl-gkmdruxos0-zk-2dee0d8-ohwork1.com.tw","hostAliases":[{"ip":"192.168.56.130","hostnames":["ohara.com.tw"]},{"ip":"192.168.56.133","hostnames":["ohwork1.com.tw"]},{"ip":"192.168.56.134","hostnames":["ohwork2.com.tw"]},{"ip":"192.168.56.135","hostnames":["ohwork3.com.tw"]}],"subdomain":"default","nodeSelector":{"kubernetes.io/hostname":"ohwork1.com.tw"},"containers":[{"name":"ohara","image":"oharastream/zookeeper:0.6.0","ports":[{"containerPort":45391,"hostPort":45391},{"containerPort":13891,"hostPort":13891},{"containerPort":49730,"hostPort":49730}],"command":[],"args":[],"imagePullPolicy":"IfNotPresent","env":[{"name":"ZK_ID","value":"0"},{"name":"ZK_ELECTION_PORT","value":"49730"},{"name":"ZK_CLIENT_PORT","value":"45391"},{"name":"ZK_SERVERS","value":"ohwork1.com.tw"},{"name":"ZK_PEER_PORT","value":"13891"}]}]}}
2019-08-12 08:55:11,394 ERROR [Configurator-system-akka.actor.default-dispatcher-12] configurator.Configurator$$anonfun$exceptionHandler$1(118): Request to http://192.168.56.130:5000/v0/zookeepers with HttpEntity.Strict(application/json,{"name":"gkmdruxos0","clientPort":45391,"peerPort":13891,"electionPort":49730,"nodeNames":["ohwork1.com.tw"]}) could not be handled normally
java.lang.RuntimeException: Pod "k8soccl-gkmdruxos0-zk-2dee0d8" is invalid: spec.hostname: Invalid value: "k8soccl-gkmdruxos0-zk-2dee0d8-ohwork1.com.tw": a DNS-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', regex used for validation is 'a-z0-9?')
at com.island.ohara.agent.k8s.K8SClient$$anon$1.$anonfun$unmarshal$1(K8SClient.scala:342)
at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
at akka.http.scaladsl.util.FastFuture$.$anonfun$transformWith$1(FastFuture.scala:37)
at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension0(FastFuture.scala:37)
at akka.http.scaladsl.util.FastFuture$FulfilledFuture.transformWith(FastFuture.scala:86)
at scala.concurrent.Future.flatMap(Future.scala:306)
at scala.concurrent.Future.flatMap$(Future.scala:306)
at akka.http.scaladsl.util.FastFuture$FulfilledFuture.flatMap(FastFuture.scala:77)
at com.island.ohara.agent.k8s.K8SClient$$anon$1.com$island$ohara$agent$k8s$K8SClient$$anon$$unmarshal(K8SClient.scala:341)
at com.island.ohara.agent.k8s.K8SClient$$anon$1$$anon$2.$anonfun$run$7(K8SClient.scala:309)
at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-08-12 08:55:11,715 ERROR [Configurator-system-akka.actor.default-dispatcher-2] configurator.Configurator$$anonfun$exceptionHandler$1(113): Request to http://192.168.56.130:5000/v0/brokers with HttpEntity.Strict(application/json,{"name":"b7n8q0d6oh","nodeNames":["ohwork1.com.tw","ohwork2.com.tw","ohwork3.com.tw"],"clientPort":63265,"exporterPort":55329,"jmxPort":54700}) is wrong
java.lang.IllegalArgumentException: You didn't specify the zk cluster for bk cluster:b7n8q0d6oh, and there is no default zk cluster
at com.island.ohara.configurator.route.BrokerRoute$.$anonfun$apply$11(BrokerRoute.scala:56)
at scala.Option.getOrElse(Option.scala:138)
at com.island.ohara.configurator.route.BrokerRoute$.$anonfun$apply$5(BrokerRoute.scala:51)
at com.island.ohara.configurator.route.RouteUtils$.$anonfun$basicRouteOfCluster$6(RouteUtils.scala:152)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-08-12 08:55:11,982 ERROR [Configurator-system-akka.actor.default-dispatcher-4] configurator.Configurator$$anonfun$exceptionHandler$1(113): Request to http://192.168.56.130:5000/v0/workers with HttpEntity.Strict(application/json,{"name":"ohtest","jmxPort":43721,"clientPort":5678,"nodeNames":["ohwork1.com.tw","ohwork2.com.tw","ohwork3.com.tw"],"jars":[],"groupId":"2pj3vcbpra","configTopicName":"v4kftaw0gw","offsetTopicName":"s180wk926d","statusTopicName":"qh8u8p6smh"}) is wrong
java.lang.IllegalArgumentException: You didn't specify the bk cluster for wk cluster:ohtest, and there is no default bk cluster
at com.island.ohara.configurator.route.WorkerRoute$.$anonfun$apply$22(WorkerRoute.scala:82)
at scala.Option.getOrElse(Option.scala:138)
at com.island.ohara.configurator.route.WorkerRoute$.$anonfun$apply$16(WorkerRoute.scala:77)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-08-12 08:55:13,313 INFO [main] configurator.Configurator$(428): Current data size:3
2019-08-12 08:55:15,314 INFO [main] configurator.Configurator$(428): Current data size:3
2019-08-12 08:55:17,315 INFO [main] configurator.Configurator$(428): Current data size:3
2019-08-12 08:55:19,316 INFO [main] configurator.Configurator$(428): Current data size:3
2019-08-12 08:55:21,317 INFO [main] configurator.Configurator$(428): Current data size:3
2019-08-12 08:55:23,318 INFO [main] configurator.Configurator$(428): Current dat

[chia7712]

FYI @jackyoh

[jackyoh]

從 log 看起來是 K8S 環境 dns 的問題，請參考文件如下連結：
https://ohara.readthedocs.io/en/latest/user_guide.html?highlight=kubectl%20apply
項目為：How to use Kubernetes in Ohara?

kubectl create -f dns-service.yaml

dns-service.yaml 可以在以下連結找到：
https://github.com/oharastream/ohara/tree/master/kubernetes

[louie5486]

確認環境早已安裝dns service，再次測試，問題還是一樣.

[jackyoh]

使用以下指令，貼結果給我看看:
$ kubectl get nodes

[louie5486]

[jackyoh]

看有沒有辦法把 .com.tw 的 host name 移除掉測試看看，我想確認是否是 k8s 版本的 bug

[louie5486]

已將k8s reset，並且將所有node的hostname 皆改為short hostname.
重新進行測試後，該問題已排除，謝謝jack協助~

[konekoya]

If the issue has been resolved, either @jackyoh or @louie5486 could you please close the issue :)

[jackyoh]

有關於 .com.tw 的部份，我還會另外去開一個 issue 來處裡這個部份的問題。我先把這個 issue 關閉。