Description
OS and version:
centos7.3
Docker:
Client:
Version: 1.13.1
API version: 1.26
Package version: docker-1.13.1-63.git94f4240.el7.centos.x86_64
Go version: go1.9.4
Git commit: 94f4240/1.13.1
Built: Fri May 18 15:44:33 2018
OS/Arch: linux/amd64
Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Package version: docker-1.13.1-63.git94f4240.el7.centos.x86_64
Go version: go1.9.4
Git commit: 94f4240/1.13.1
Built: Fri May 18 15:44:33 2018
OS/Arch: linux/amd64
openshift:
oc v3.9.0+191fece
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO
Server https://10.121.198.93:8443
openshift v3.9.0+71543b2-33
kubernetes v1.9.1+a0ce1bc657
new apps:che with che-server-template.yaml
now che running on openshift.
when I create workspace on che ,it was occured "create workspace fail"
This is logs:
2018-07-12 07:05:10,623[aceSharedPool-1] [WARN ] [o.e.c.a.w.s.WorkspaceManager 423] - Cannot set error status of the workspace workspace269y3vjyupp1o0su. Error is: null
2018-07-12 07:07:35,322[nio-8080-exec-4] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 329] - Starting workspace 'che/wksp-s1fb' with id 'workspace269y3vjyupp1o0su' by user 'che'
2018-07-12 07:15:42,707[aceSharedPool-2] [WARN ] [.i.k.KubernetesInternalRuntime 191] - Failed to start Kubernetes runtime of workspace workspace269y3vjyupp1o0su. Cause: null
2018-07-12 07:15:43,519[aceSharedPool-2] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 390] - Workspace 'che:wksp-s1fb' with id 'workspace269y3vjyupp1o0su' start failed
2018-07-12 07:15:43,519[aceSharedPool-2] [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 400] - null
org.eclipse.che.api.workspace.server.spi.InternalInfrastructureException: null
at org.eclipse.che.workspace.infrastructure.kubernetes.StartSynchronizer.getStartFailureNow(StartSynchronizer.java:274)
at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:186)
at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:145)
at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:366)
at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:37)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: null
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.waitMachines(KubernetesInternalRuntime.java:251)
at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:183)
... 7 common frames omitted
2018-07-12 07:15:43,575[aceSharedPool-2] [WARN ] [o.e.c.a.w.s.WorkspaceManager 423] - Cannot set error status of the workspace workspace269y3vjyupp1o0su. Error is: null
Can I get a guide in here?
zeyuanpinghe
All 13 comments
@kingkeivn Could you specify a json of a workspace config which you try to start? I guess null error may be produced by invalid Kubernetes objects in a recipe.
If they are correct then maybe OS events could help us to clarify a cause of the fault.
sleshchenko
on 12 Jul 2018
@sleshchenko Thank you for answering this issue.
And this is the workspace config which I try to start.
{
"environments": {
"default": {
"machines": {
"dev-machine": {
"attributes": {
"memoryLimitBytes": "2147483648"
},
"servers": {
"tomcat8-debug": {
"attributes": {},
"port": "8000",
"protocol": "http"
},
"codeserver": {
"attributes": {},
"port": "9876",
"protocol": "http"
},
"tomcat8": {
"attributes": {},
"port": "8080",
"protocol": "http"
}
},
"volumes": {},
"installers": [
"org.eclipse.che.exec",
"org.eclipse.che.terminal",
"org.eclipse.che.ws-agent"
],
"env": {
"CHE_MACHINE_NAME": "dev-machine"
}
}
},
"recipe": {
"type": "dockerimage",
"content": "eclipse/ubuntu_jdk8"
}
}
},
"defaultEnv": "default",
"projects": [],
"name": "wksp-s1fb",
"commands": [
{
"commandLine": "mvn clean install -f ${current.project.path}",
"name": "build",
"attributes": {
"goal": "Build",
"previewUrl": ""
},
"type": "mvn"
}
],
"links": []
}
zeyuanpinghe
on 13 Jul 2018
@kingkeivn Workspace configuration looks just fine
new apps:che with che-server-template.yaml
Could you provide a full command that you used for application creation?
Which events are present in OpenShift web console during workspace start?
sleshchenko
on 13 Jul 2018
@kingkeivn ideally, can you share deploymentconfig as yaml? I suspect there might be some bad envs there. Applications> Deployments > che > Actions > Edit Yaml and copy. Thanks
ghost
on 13 Jul 2018
@eivantsov
Thanks.
This is the content in yaml file.
apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
creationTimestamp: '2018-07-05T12:40:04Z'
generation: 12
labels:
app: che
template: che
name: che
namespace: myproject
resourceVersion: '914011'
selfLink: /apis/apps.openshift.io/v1/namespaces/myproject/deploymentconfigs/che
uid: 8a717e59-8050-11e8-abb4-0cda411d170a
spec:
replicas: 1
revisionHistoryLimit: 2
selector:
app: che
strategy:
activeDeadlineSeconds: 21600
recreateParams:
timeoutSeconds: 600
resources: {}
type: Recreate
template:
metadata:
creationTimestamp: null
labels:
app: che
spec:
containers:
- env:
- name: OPENSHIFT_KUBE_PING_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: CHE_MULTIUSER
value: 'false'
- name: CHE_HOST
value: 'che-${NAMESPACE}.'
- name: CHE_PORT
value: '8899'
- name: CHE_API
value: 'http://che-${NAMESPACE}./api'
- name: CHE_WEBSOCKET_ENDPOINT
value: 'ws://che-${NAMESPACE}./api/websocket'
- name: CHE_DEBUG_SERVER
value: 'false'
- name: CHE_INFRASTRUCTURE_ACTIVE
value: openshift
- name: CHE_INFRA_KUBERNETES_BOOTSTRAPPER_BINARY__URL
value: >-
http://che-${NAMESPACE}.10.121.198.93.nip.io/agent-binaries/linux_amd64/bootstrapper/bootstrapper
- name: CHE_INFRA_KUBERNETES_MACHINE__START__TIMEOUT__MIN
value: '5'
- name: CHE_INFRA_KUBERNETES_MASTER__URL
- name: CHE_INFRA_KUBERNETES_OAUTH__TOKEN
- name: CHE_INFRA_KUBERNETES_USERNAME
- name: CHE_INFRA_KUBERNETES_PASSWORD
- name: CHE_INFRA_OPENSHIFT_PROJECT
value: '${NAMESPACE}'
- name: CHE_INFRA_KUBERNETES_PVC_STRATEGY
value: unique
- name: CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS
value: 'false'
- name: CHE_INFRA_OPENSHIFT_TLS__ENABLED
value: 'false'
- name: CHE_INFRA_KUBERNETES_TRUST__CERTS
value: 'false'
- name: CHE_LOCAL_CONF_DIR
value: /etc/conf
- name: CHE_LOGS_DIR
value: /data/logs
- name: CHE_LOG_LEVEL
value: INFO
- name: CHE_KEYCLOAK_AUTH__SERVER__URL
value: '${PROTOCOL}://keycloak-${NAMESPACE}.${ROUTING_SUFFIX}/auth'
- name: CHE_INFRA_OPENSHIFT_OAUTH__IDENTITY__PROVIDER
value: 'NULL'
- name: CHE_OAUTH_GITHUB_CLIENTID
- name: CHE_OAUTH_GITHUB_CLIENTSECRET
- name: CHE_PREDEFINED_STACKS_RELOAD__ON__START
value: 'true'
- name: JAVA_OPTS
value: >-
-XX:MaxRAMFraction=2 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10
-XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4
-XX:AdaptiveSizePolicyWeight=90 -XX:+UnlockExperimentalVMOptions
-XX:+UseCGroupMemoryLimitForHeap
-Dsun.zip.disableMemoryMapping=true -Xms20m
- name: CHE_WORKSPACE_AUTO_START
value: 'false'
- name: CHE_INFRA_KUBERNETES_PVC_QUANTITY
value: 1Gi
- name: PROTOCOL
value: http
- name: ROUTING_SUFFIX
value: 10.121.198.90.nip.io
- name: OPENSHIFT_IDENTITY_PROVIDER_CERTIFICATE
valueFrom:
secretKeyRef:
key: ca.crt
name: openshift-identity-provider
optional: true
- name: CHE_WORKSPACE_FEATURE_API
value: 'http://feature-api-service:3000'
image: 'docker.io/eclipse/che-server:nightly'
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
httpGet:
path: /api/system/state
port: 8080
scheme: HTTP
initialDelaySeconds: 50
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
name: che
ports:
- containerPort: 8080
name: http
protocol: TCP
- containerPort: 8000
name: http-debug
protocol: TCP
- containerPort: 8888
name: jgroups-ping
protocol: TCP
readinessProbe:
failureThreshold: 5
httpGet:
path: /api/system/state
port: 8080
scheme: HTTP
initialDelaySeconds: 25
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
memory: 1Gi
requests:
memory: 256Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: che
serviceAccountName: che
terminationGracePeriodSeconds: 360
test: false
triggers:
- type: ConfigChange
status:
availableReplicas: 0
conditions:
- lastTransitionTime: '2018-07-13T00:44:05Z'
lastUpdateTime: '2018-07-13T00:44:05Z'
message: Deployment config does not have minimum availability.
status: 'False'
type: Available
- lastTransitionTime: '2018-07-13T05:38:49Z'
lastUpdateTime: '2018-07-13T05:38:49Z'
message: replication controller "che-12" is progressing
reason: ReplicationControllerUpdated
status: 'True'
type: Progressing
details:
causes:
- type: Manual
message: manual change
latestVersion: 12
observedGeneration: 12
replicas: 0
unavailableReplicas: 0
updatedReplicas: 0
@kingkeivn somehow a few envs are messed up:
- name: CHE_HOST
value: 'che-${NAMESPACE}.'
- name: CHE_PORT
value: '8899'
- name: CHE_API
value: 'http://che-${NAMESPACE}./api'
- name: CHE_WEBSOCKET_ENDPOINT
value: 'ws://che-${NAMESPACE}./api/websocket'
Can you update urls, so that you have 10.121.198.90.nip.io after ${NAMESPACE}.
Also, please, use latest tag singe nightly build may be broken at this very moment
image: 'docker.io/eclipse/che-server:latest'
So, update your dc/che and let's see if it helps. Maybe I missed other bad urls, take a precise look at your deploymentConfig.
How did you deploy? According to docs? Templates seem to be just fine https://github.com/eclipse/che/blob/master/deploy/openshift/templates/che-server-template.yaml#L81 and in some envs routing suffix was inserted.
ghost
on 13 Jul 2018
@sleshchenko I just redeployed the project che ,but it can't continue work.
zeyuanpinghe
on 13 Jul 2018
Do you see a pod being created when you start a workspace? Or it immediately fails?
oc get events in this project please
ghost
on 13 Jul 2018
@eivantsov I need to specify this version explicitly of image in this yaml ?
zeyuanpinghe
on 13 Jul 2018
@kingkeivn yes
image: 'docker.io/eclipse/che-server:latest'
And make sure you update envs with correct routing suffix
ghost
on 13 Jul 2018
@eivantsov
I deploying che success with your advice.
Now,che create workspace success.
But,I fetch logs with :oc logs -f che-1-rd97t
And,it show me a error :
2018-07-13 06:29:08,601[equestProcessor] [ERROR] [l.ErrorInstallerLogEventLogger 42] - Installer org.eclipse.che.ws-agent error from machine=dev-machine owner=che env=default workspace=workspacem5n5qqp5cvdshd9w text=Debugger failed to attach: handshake failed - connection prematurally closed time=2018-07-13T06:29:08.598545074Z
zeyuanpinghe
on 13 Jul 2018
@kingkeivn I would say that it is well-known bug. And this line should be just ignored and it won't produce any errors.
sleshchenko
on 13 Jul 2018
@sleshchenko yes,it not produce any errors. Thank you .
@eivantsov Thank you .
And I will close this issue.
zeyuanpinghe
on 13 Jul 2018
Related issues
SkorikSergey
路
3Comments
AndrienkoAleksandr
路
3Comments
dmytro-ndp
路
3Comments
sleshchenko
路
3Comments
vanzhiganov
路
3Comments