Tf-operator: ERROR handle object: patching object from cluster: merging object with existing state: unable to recognize "/var/folders/tl/zzfcr4zs53vgnpqqjq4n08sh0000gn/T/ksonnet-mergepatch020443124": no matches for kind "TFJob" in version "kubeflow.org/v1beta1"
I am trying to follow the guide here: https://www.kubeflow.org/docs/gke/gcp-e2e/
However, when I am trying to apply my TFJob chart on cluster it is giving me above error:
ks apply default -c train
ERROR handle object: patching object from cluster: merging object with existing state: unable to recognize "/var/folders/tl/zzfcr4zs53vgnpqqjq4n08sh0000gn/T/ksonnet-mergepatch020443124": no matches for kind "TFJob" in version "kubeflow.org/v1beta1"
I have tfjobs in my custom resources:
$kubectl get crd
NAME CREATED AT
backendconfigs.cloud.google.com 2019-04-16T06:37:58Z
certificates.certmanager.k8s.io 2019-04-16T06:39:42Z
cloudendpoints.ctl.isla.solutions 2019-04-16T06:39:49Z
clusterissuers.certmanager.k8s.io 2019-04-16T06:39:42Z
compositecontrollers.metacontroller.k8s.io 2019-04-16T06:40:41Z
controllerrevisions.metacontroller.k8s.io 2019-04-16T06:40:41Z
decoratorcontrollers.metacontroller.k8s.io 2019-04-16T06:40:41Z
issuers.certmanager.k8s.io 2019-04-16T06:39:42Z
notebooks.kubeflow.org 2019-04-16T06:40:48Z
pytorchjobs.kubeflow.org 2019-04-16T06:41:06Z
scalingpolicies.scalingpolicy.kope.io 2019-04-16T06:38:27Z
scheduledworkflows.kubeflow.org 2019-04-16T06:40:56Z
studyjobs.kubeflow.org 2019-04-16T06:40:31Z
tfjobs.kubeflow.org 2019-04-16T06:41:13Z
viewers.kubeflow.org 2019-04-16T06:40:57Z
workflows.argoproj.io 2019-04-16T06:39:27Z
Here is the output yaml:
$kubectl get crd tfjobs.kubeflow.org -o yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
annotations:
ksonnet.io/managed: '{"pristine":"H4sIAAAAAAAA/6STwW7bMAyG73sMnl0P2S6DbkWHAeuhC5qiOxQ50DbtqJFEgaKyDoHffZDtJguKoYf5JJLiR/MndQSM9pEkWQ5gikEvSqGYqd5/SbXlj4dVQ4orqGBvQwcGbnJS9veUOEtLX6m3wWoBVOBJsUNFMEdw2JBL5bRPHAJpgbXsIwcKCga0v3rm5oojCSoLjBUE9DRFnrlJ9T431Dv+VbMMJZoitYWHXTcVRLcWG5Tkhl32IYF5OsLt5sfdGnUHBuqkqDnVLYc5IT1drcy21t+R4FRso6jFnLwGkooNpdwl6rWzuhXCwnqwnpKij2fS9fAXpyvUcVvBIJwjGLjoZs6Z1ZlVffh2yw1UEF0WdCcRoIJkw5AdyqtvkqLlqcpdoURsqSsXcyPLVCby3D6Y4zhWcEBnu+nPS4gjhev198fPm3ZHfppXlDIJtUvuovWlV/t7is62uInUprfxm52l/q1b5qTp7PHF+uzBrCrwNpzOi2xlngMJjOWf15t3WO/k/2TZk/wPY/kqOJweyfk9LL5575YdOEcTyYE6MCqZKkjKggPNZlmui/uf/n2/R5do3I7jhz8AAAD//wEAAP//+qGEoa0DAAA="}'
kubecfg.ksonnet.io/garbage-collect-tag: gc-tag
creationTimestamp: 2019-04-16T06:41:13Z
generation: 1
labels:
app.kubernetes.io/deploy-manager: ksonnet
ksonnet.io/component: tf-job-operator
name: tfjobs.kubeflow.org
resourceVersion: "1905"
selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/tfjobs.kubeflow.org
uid: a0a00691-6012-11e9-9745-42010a80009b
spec:
additionalPrinterColumns:
- JSONPath: .status.conditions[-1:].type
name: State
type: string
- JSONPath: .metadata.creationTimestamp
name: Age
type: date
group: kubeflow.org
names:
kind: TFJob
listKind: TFJobList
plural: tfjobs
singular: tfjob
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
properties:
spec:
properties:
tfReplicaSpecs:
properties:
Chief:
properties:
replicas:
maximum: 1
minimum: 1
type: integer
PS:
properties:
replicas:
minimum: 1
type: integer
Worker:
properties:
replicas:
minimum: 1
type: integer
version: v1beta1
versions:
- name: v1beta1
served: true
storage: true
- name: v1beta2
served: true
storage: false
status:
acceptedNames:
kind: TFJob
listKind: TFJobList
plural: tfjobs
singular: tfjob
conditions:
- lastTransitionTime: 2019-04-16T06:41:13Z
message: no conflicts found
reason: NoConflicts
status: "True"
type: NamesAccepted
- lastTransitionTime: null
message: the initial names have been accepted
reason: InitialNamesAccepted
status: "True"
type: Established
storedVersions:
- v1beta1
I have pod up and running:
$kubectl get pods -n kubeflow | grep tf
jupyter-web-app-847f4f465c-tf7xh 1/1 Running 0 5d
tf-job-dashboard-ffd74f79-7gzfj 1/1 Running 0 5d
tf-job-operator-d979d98dd-nf4zn 1/1 Running 2 5d
However, the logs inside pods seems to giving connection refused error:
$kubectl logs tf-job-operator-d979d98dd-nf4zn
{"filename":"app/server.go:66","level":"info","msg":"EnvKubeflowNamespace not set, use default namespace","time":"2019-04-18T06:31:35Z"}
{"filename":"app/server.go:70","level":"info","msg":"Using cluster scoped operator","time":"2019-04-18T06:31:35Z"}
{"filename":"app/server.go:76","level":"info","msg":"[API Version: v1beta2 Version: v0.1.0-alpha Git SHA: aa322c7 Go Version: go1.9.2 Go OS/Arch: linux/amd64]","time":"2019-04-18T06:31:35Z"}
W0418 06:31:35.309285 1 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
{"filename":"app/server.go:196","level":"error","msg":"Get https://10.39.240.1:443/apis/kubeflow.org/v1beta2/tfjobs: dial tcp 10.39.240.1:443: getsockopt: connection refused","time":"2019-04-18T06:31:35Z"}
{"filename":"tensorflow/controller.go:115","level":"info","msg":"Creating TFJob controller","time":"2019-04-18T06:31:35Z"}
{"filename":"tensorflow/controller.go:122","level":"info","msg":"Creating Job controller","time":"2019-04-18T06:31:35Z"}
I0418 06:31:35.317252 1 leaderelection.go:185] attempting to acquire leader lease default/tf-operator...
E0418 06:31:35.318309 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Pod: Get https://10.39.240.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:35.318310 1 reflector.go:205] github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:118: Failed to list *unstructured.Unstructured: Get https://10.39.240.1:443/apis/kubeflow.org/v1beta2/tfjobs?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:35.318631 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://10.39.240.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:35.318752 1 leaderelection.go:234] error retrieving resource lock default/tf-operator: Get https://10.39.240.1:443/api/v1/namespaces/default/endpoints/tf-operator: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:36.320878 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Pod: Get https://10.39.240.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:36.321504 1 reflector.go:205] github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:118: Failed to list *unstructured.Unstructured: Get https://10.39.240.1:443/apis/kubeflow.org/v1beta2/tfjobs?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:36.322574 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://10.39.240.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:37.323065 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Pod: Get https://10.39.240.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:37.324063 1 reflector.go:205] github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:118: Failed to list *unstructured.Unstructured: Get https://10.39.240.1:443/apis/kubeflow.org/v1beta2/tfjobs?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:37.325146 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://10.39.240.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:38.325121 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Pod: Get https://10.39.240.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:38.325984 1 reflector.go:205] github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:118: Failed to list *unstructured.Unstructured: Get https://10.39.240.1:443/apis/kubeflow.org/v1beta2/tfjobs?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:38.327017 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://10.39.240.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:39.366671 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Pod: Get https://10.39.240.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:39.366787 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://10.39.240.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:39.366786 1 reflector.go:205] github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:118: Failed to list *unstructured.Unstructured: Get https://10.39.240.1:443/apis/kubeflow.org/v1beta2/tfjobs?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:40.368769 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Pod: Get https://10.39.240.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:40.369704 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://10.39.240.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:40.370743 1 reflector.go:205] github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:118: Failed to list *unstructured.Unstructured: Get https://10.39.240.1:443/apis/kubeflow.org/v1beta2/tfjobs?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:40.497723 1 leaderelection.go:234] error retrieving resource lock default/tf-operator: Get https://10.39.240.1:443/api/v1/namespaces/default/endpoints/tf-operator: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:41.370844 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Pod: Get https://10.39.240.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:41.371837 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://10.39.240.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:41.373224 1 reflector.go:205] github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:118: Failed to list *unstructured.Unstructured: Get https://10.39.240.1:443/apis/kubeflow.org/v1beta2/tfjobs?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:42.372645 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Pod: Get https://10.39.240.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:42.373501 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://10.39.240.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:42.375010 1 reflector.go:205] github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:118: Failed to list *unstructured.Unstructured: Get https://10.39.240.1:443/apis/kubeflow.org/v1beta2/tfjobs?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:43.374542 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Pod: Get https://10.39.240.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:43.375397 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://10.39.240.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:43.376672 1 reflector.go:205] github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:118: Failed to list *unstructured.Unstructured: Get https://10.39.240.1:443/apis/kubeflow.org/v1beta2/tfjobs?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:44.376748 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Pod: Get https://10.39.240.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:44.377520 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://10.39.240.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:44.378439 1 reflector.go:205] github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:118: Failed to list *unstructured.Unstructured: Get https://10.39.240.1:443/apis/kubeflow.org/v1beta2/tfjobs?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:45.379031 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Pod: Get https://10.39.240.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
E0418 06:31:45.379849 1 reflector.go:205] github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://10.39.240.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.39.240.1:443: getsockopt: connection refused
Other pods seems to be working fine:
$kubectl logs tf-job-dashboard-ffd74f79-7gzfj
2019/04/16 06:42:15 Dashboard available at /tfjobs/ui/ on port :8080
or
$kubectl logs jupyter-web-app-847f4f465c-tf7xh
* Serving Flask app "kubeflow_jupyter.default.app" (lazy loading)
* Environment: production
WARNING: Do not use the development server in a production environment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
10.36.0.10 - - [16/Apr/2019 06:43:55] "GET / HTTP/1.1" 200 -
10.36.0.10 - - [16/Apr/2019 06:43:56] "GET /static/js/notebooks.js HTTP/1.1" 200 -
10.36.0.10 - - [16/Apr/2019 06:43:56] "GET /static/js/dialog.js HTTP/1.1" 200 -
10.36.0.10 - - [16/Apr/2019 06:43:56] "GET /static/css/main.css HTTP/1.1" 200 -
10.36.0.10 - - [16/Apr/2019 06:43:56] "GET /static/css/dialog.css HTTP/1.1" 200 -
10.36.0.10 - - [16/Apr/2019 06:43:58] "GET /api/namespaces/kubeflow/notebooks HTTP/1.1" 200 -
10.36.0.10 - - [16/Apr/2019 06:43:59] "GET /api/namespaces/kubeflow/notebooks HTTP/1.1" 200 -
10.36.0.10 - - [16/Apr/2019 06:44:00] "GET /api/namespaces/kubeflow/notebooks HTTP/1.1" 200 -
kinquestion
d34th4ck3r
All 9 comments
Issue-Label Bot is automatically applying the label community/question to this issue, with a confidence of 0.67. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.
![issue-label-bot[bot] picture](https://avatars.githubusercontent.com/in/27079?v=4&s=40)
issue-label-bot[bot]
on 21 Apr 2019
👎1
👍1
I think you may register TFJob v1beta2, while the config is v1beta1, then it is not found.
gaocegege
on 21 Apr 2019
I tried changing the apiVersion from kubeflow.org/v1beta1 to kubeflow.org/v1beta2 in my train.jsonnet but that gave the same error:
$ks apply default -c train
ERROR handle object: patching object from cluster: merging object with existing state: unable to recognize "/var/folders/tl/zzfcr4zs53vgnpqqjq4n08sh0000gn/T/ksonnet-mergepatch059097723": no matches for kind "TFJob" in version "kubeflow.org/v1beta2"
What's the output of kubectl describe crd tfjobs.kubeflow.org
gaocegege
on 22 Apr 2019
Here:
$kubectl describe crd tfjobs.kubeflow.org
Name: tfjobs.kubeflow.org
Namespace:
Labels: app.kubernetes.io/deploy-manager=ksonnet
ksonnet.io/component=tf-job-operator
Annotations: ksonnet.io/managed={"pristine":"H4sIAAAAAAAA/6STwW7bMAyG73sMnl0P2S6DbkWHAeuhC5qiOxQ50DbtqJFEgaKyDoHffZDtJguKoYf5JJLiR/MndQSM9pEkWQ5gikEvSqGYqd5/SbXlj4dVQ4orqGBvQwcGbnJS9veUOEtLX6m3wWoBVOBJsUNFMEdw2JBL...
kubecfg.ksonnet.io/garbage-collect-tag=gc-tag
API Version: apiextensions.k8s.io/v1beta1
Kind: CustomResourceDefinition
Metadata:
Creation Timestamp: 2019-04-16T06:41:13Z
Generation: 1
Resource Version: 1905
Self Link: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/tfjobs.kubeflow.org
UID: a0a00691-6012-11e9-9745-42010a80009b
Spec:
Additional Printer Columns:
JSON Path: .status.conditions[-1:].type
Name: State
Type: string
JSON Path: .metadata.creationTimestamp
Name: Age
Type: date
Group: kubeflow.org
Names:
Kind: TFJob
List Kind: TFJobList
Plural: tfjobs
Singular: tfjob
Scope: Namespaced
Subresources:
Status:
Validation:
Open APIV 3 Schema:
Properties:
Spec:
Properties:
Tf Replica Specs:
Properties:
Chief:
Properties:
Replicas:
Maximum: 1
Minimum: 1
Type: integer
PS:
Properties:
Replicas:
Minimum: 1
Type: integer
Worker:
Properties:
Replicas:
Minimum: 1
Type: integer
Version: v1beta1
Versions:
Name: v1beta1
Served: true
Storage: true
Name: v1beta2
Served: true
Storage: false
Status:
Accepted Names:
Kind: TFJob
List Kind: TFJobList
Plural: tfjobs
Singular: tfjob
Conditions:
Last Transition Time: 2019-04-16T06:41:13Z
Message: no conflicts found
Reason: NoConflicts
Status: True
Type: NamesAccepted
Last Transition Time: <nil>
Message: the initial names have been accepted
Reason: InitialNamesAccepted
Status: True
Type: Established
Stored Versions:
v1beta1
Events: <none>
It seems that the v1alpha1 is already registered. I will have a try. But the operator cannot visit apiserver, from your case.
gaocegege
on 24 Apr 2019
How did it go?
d34th4ck3r
on 26 Apr 2019
Is this issue resolved?
Btw we have now deprecated v1beta1 in the latest version.
richardsliu
on 7 Jun 2019
I am closing the issue since it is stale. But feel free to leave comments if you still have problems.
gaocegege
on 29 Oct 2019
Was this page helpful?
0 / 5 - 0 ratings
Related issues
izhaomeng
路
9Comments
yph152
路
10Comments
Jeffwan
路
6Comments
jlewi
路
9Comments
chaoleili
路
8Comments