Tf-operator: Can not create tfjob using examples/v1beta1/dist-mnist/tf_job_mnist.yaml in self-created k8s cluster and tf-operator

[izhaomeng]

I can not link gci , so i use the image jackfantasy/tf-dist-mnist-test:1.0

The k8s is installed on some vm nodes using the flannel network plugin and the kubeflow/tf-operator with version 0.5.0 is installed by ksonnet.

The tf-job yaml is listed below:

apiVersion: "kubeflow.org/v1beta1"
kind: "TFJob"
metadata:
  name: "dist-mnist-for-e2e-test"
spec:
  tfReplicaSpecs:
    PS:
      replicas: 2
      restartPolicy: Never
      template:
        spec:
          containers:
            - name: tensorflow
              image: jackfantasy/tf-dist-mnist-test:1.0
    Worker:
      replicas: 4
      restartPolicy: Never
      template:
        spec:
          containers:
            - name: tensorflow
              image: jackfantasy/tf-dist-mnist-test:1.0

Start the job by script:

kubectl apply -f mnist_multi_cpu.yaml -n zhaomeng

The script "kubectl get all -n zhaomeng " output is like this:

NAME                                    READY   STATUS    RESTARTS   AGE
pod/dist-mnist-for-e2e-test-ps-0        1/1     Running   0          18s
pod/dist-mnist-for-e2e-test-ps-1        1/1     Running   0          17s
pod/dist-mnist-for-e2e-test-worker-0    1/1     Running   0          18s
pod/dist-mnist-for-e2e-test-worker-1    1/1     Running   0          18s
pod/dist-mnist-for-e2e-test-worker-2    1/1     Running   0          18s
pod/dist-mnist-for-e2e-test-worker-3    1/1     Running   0          18s
pod/tf-job-dashboard-74f8c769c9-6xv5w   1/1     Running   0          7h17m
pod/tf-job-operator-7b86667c9b-xnhfg    1/1     Running   2          45h

NAME                                       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/dist-mnist-for-e2e-test-ps-0       ClusterIP   None           <none>        2222/TCP       17s
service/dist-mnist-for-e2e-test-ps-1       ClusterIP   None           <none>        2222/TCP       16s
service/dist-mnist-for-e2e-test-worker-0   ClusterIP   None           <none>        2222/TCP       18s
service/dist-mnist-for-e2e-test-worker-1   ClusterIP   None           <none>        2222/TCP       18s
service/dist-mnist-for-e2e-test-worker-2   ClusterIP   None           <none>        2222/TCP       18s
service/dist-mnist-for-e2e-test-worker-3   ClusterIP   None           <none>        2222/TCP       18s
service/tf-job-dashboard                   ClusterIP   10.68.128.46   <none>        80/TCP         45h
service/tf-job-dashboard-clusterip         ClusterIP   10.68.173.74   <none>        80/TCP         26h
service/tf-job-dashboard-nodeport          NodePort    10.68.218.52   <none>        80:39861/TCP   26h

The script " kubectl get all -n zhaomeng" output is like this:

NAME                                    READY   STATUS    RESTARTS   AGE
pod/dist-mnist-for-e2e-test-ps-0        0/1     Error     0          2m57s
pod/tf-job-dashboard-74f8c769c9-6xv5w   1/1     Running   0          7h19m
pod/tf-job-operator-7b86667c9b-xnhfg    1/1     Running   2          45h

NAME                                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/dist-mnist-for-e2e-test-ps-0   ClusterIP   None           <none>        2222/TCP       2m56s
service/tf-job-dashboard               ClusterIP   10.68.128.46   <none>        80/TCP         45h
service/tf-job-dashboard-clusterip     ClusterIP   10.68.173.74   <none>        80/TCP         26h
service/tf-job-dashboard-nodeport      NodePort    10.68.218.52   <none>        80:39861/TCP   26h

The script "kubectl describe service dist-mnist-for-e2e-test-ps-0 -n zhaomeng" output is like this:

Name:              dist-mnist-for-e2e-test-ps-0
Namespace:         zhaomeng
Labels:            group-name=kubeflow.org
                   tf-job-name=dist-mnist-for-e2e-test
                   tf-replica-index=0
                   tf-replica-type=ps
Annotations:       <none>
Selector:          group-name=kubeflow.org,tf-job-name=dist-mnist-for-e2e-test,tf-replica-index=0,tf-replica-type=ps
Type:              ClusterIP
IP:                None
Port:              tfjob-port  2222/TCP
TargetPort:        2222/TCP
Endpoints:         <none>
Session Affinity:  None
Events:            <none>

The script "kubectl describe pod dist-mnist-for-e2e-test-ps-0 -n zhaomeng" output is like this:

Name:           dist-mnist-for-e2e-test-ps-0
Namespace:      zhaomeng
Node:           10.13.33.38/10.13.33.38
Start Time:     Fri, 19 Apr 2019 19:17:11 +0800
Labels:         group-name=kubeflow.org
                tf-job-name=dist-mnist-for-e2e-test
                tf-replica-index=0
                tf-replica-type=ps
Annotations:    <none>
Status:         Failed
IP:             10.244.1.110
Controlled By:  TFJob/dist-mnist-for-e2e-test
Containers:
  tensorflow:
    Container ID:   docker://6332e5a4e9fe6b8b3c8b52af57f81aac0e85c25ec11263c28632726a9b427324
    Image:          jackfantasy/tf-dist-mnist-test:1.0
    Image ID:       docker-pullable://jackfantasy/tf-dist-mnist-test@sha256:06c592199dfa7a28c8cea2d1f93070750c7d33c81f0a151a65bf4f8e7a5696f9
    Port:           2222/TCP
    Host Port:      0/TCP
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 19 Apr 2019 19:17:12 +0800
      Finished:     Fri, 19 Apr 2019 19:17:34 +0800
    Ready:          False
    Restart Count:  0
    Environment:
      TF_CONFIG:  {"cluster":{"ps":["dist-mnist-for-e2e-test-ps-0.zhaomeng.svc:2222","dist-mnist-for-e2e-test-ps-1.zhaomeng.svc:2222"],"worker":["dist-mnist-for-e2e-test-worker-0.zhaomeng.svc:2222","dist-mnist-for-e2e-test-worker-1.zhaomeng.svc:2222","dist-mnist-for-e2e-test-worker-2.zhaomeng.svc:2222","dist-mnist-for-e2e-test-worker-3.zhaomeng.svc:2222"]},"task":{"type":"ps","index":0},"environment":"cloud"}
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-hx2dr (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-hx2dr:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-hx2dr
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type    Reason     Age   From                  Message
  ----    ------     ----  ----                  -------
  Normal  Scheduled  5m7s  default-scheduler     Successfully assigned zhaomeng/dist-mnist-for-e2e-test-ps-0 to 10.13.33.38
  Normal  Pulled     5m6s  kubelet, 10.13.33.38  Container image "jackfantasy/tf-dist-mnist-test:1.0" already present on machine
  Normal  Created    5m6s  kubelet, 10.13.33.38  Created container
  Normal  Started    5m6s  kubelet, 10.13.33.38  Started container

I can view the "dist-mnist-for-e2e-test-ps-0" log in tf-dashboard web-ui like this:

/usr/local/lib/python2.7/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Traceback (most recent call last):
  File "/var/tf_dist_mnist/dist_mnist.py", line 303, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 124, in run
    _sys.exit(main(argv))
  File "/var/tf_dist_mnist/dist_mnist.py", line 110, in main
    mnist = input_data.read_data_sets(FLAGS.data_dir, one_hot=True)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py", line 240, in read_data_sets
    source_url + TRAIN_IMAGES)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py", line 208, in maybe_download
    temp_file_name, _ = urlretrieve_with_retry(source_url)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py", line 165, in wrapped_fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py", line 190, in urlretrieve_with_retry
    return urllib.request.urlretrieve(url, filename)
  File "/usr/lib/python2.7/urllib.py", line 98, in urlretrieve
    return opener.retrieve(url, filename, reporthook, data)
  File "/usr/lib/python2.7/urllib.py", line 245, in retrieve
    fp = self.open(url, data)
  File "/usr/lib/python2.7/urllib.py", line 213, in open
    return getattr(self, name)(url)
  File "/usr/lib/python2.7/urllib.py", line 443, in open_https
    h.endheaders(data)
  File "/usr/lib/python2.7/httplib.py", line 1053, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 897, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 859, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 1270, in connect
    HTTPConnection.connect(self)
  File "/usr/lib/python2.7/httplib.py", line 836, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 557, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
IOError: [Errno socket error] [Errno -3] Temporary failure in name resolution

Anybody can help to solve the issue or explain the failed reason?

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label kind/bug to this issue, with a confidence of 0.82. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[ChanYiLin]

Hi, I am the owner of the image, jackfantasy/tf-dist-mnist-test:1.0
I created the image for my own development testing, so I modify some code inside and it may not work in your environment.

I suggest you to build your own image using our examples/v1beta1/dist-mnist

Thanks!

[gaocegege]

@izhaomeng

I think it is caused by your network environment. There are something cannot be downloaded from the internet. MNIST dataset is GFWed, maybe you should mount it into the container.

[apatsekin]

Got the same error. The reason is that containers deployed by k8s don't have internet access at all. Run on the host system: sudo iptables -P FORWARD ACCEPT . (Running same container through docker directly works fine)

[gaocegege]

@apatsekin Do you mean the pods in Kubernetes cannot communicate with each other?

[apatsekin]

@gaocegege nope, the topic issue is caused by the fact that the target python script is unable to download dataset from internet. When I tried to login into running container, it didn't have an internet access at all (although error above is DNS related). So the solution is sudo iptables -P FORWARD ACCEPT on the host.

[gaocegege]

Gotcha. Thanks for the explanation.

[whycircle]

@gaocegege 我也尝试了这个 example（https://github.com/kubeflow/tf-operator/tree/master/examples/v1/dist-mnist）直接用的是官网给出的， pod 状态是 running，但是看了 logs，worker/ps 的 logs 是这样：
/usr/local/lib/python2.7/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype fromfloattonp.floatingis deprecated. In future, it will be treated asnp.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters
请问这也是网络原因下不了 mnist 数据集嘛？

[gaocegege]

@whycircle Could you show more logs?