Che: Can't start a workspace when MetalLB exposes the Cluster ingresses
Hello,
I'm trying to deploy eclipse che 7.3.1 version on cluster bootstrapped with Rancher RKE.
The Cluster ingress is exposed with external loadbalancer IP ( with metallb)
I created offline devfile and plugin registries.
Che is deployed with helm charts and takes into account customization on
- offline plugin registry (setting CHE_SIDECAR_CONTAINERS_REGISTRY_URL)
- offline devfile registry (setting CHE_DEVFILE_REGISTRY_URL, CHE_DEVFILE_IMAGES_REGISTRY_URL)
- init and unified brokers ( adding CA CERTS)
- serverStrategy: multi-host
- multiuser: false
Che starts without error and accessible throught the ingress specified in values.yml
I created a workspace with Go stack (1.12.10) with devfile :
metadata:
name: wksp-cyjq
projects:
- name: example
source:
location: >-
https://devfile-registry-che.<fqdn>/resources/golang-example-master.zip
type: zip
components:
- mountSources: true
endpoints:
- name: 8080/tcp
port: 8080
memoryLimit: 512Mi
type: dockerimage
alias: go-cli
image: >-
repo.<fqdn>/docker/eclipse/che-golang-1.12:7.3.1
env:
- value: '/go:$(CHE_PROJECTS_ROOT)'
name: GOPATH
- value: /tmp/.cache
name: GOCACHE
- id: eclipse/che-theia/7.3.1
type: cheEditor
- id: ms-vscode/go/0.11.4
type: chePlugin
apiVersion: 1.0.0
commands:
- name: run outyet
actions:
- workdir: '${CHE_PROJECTS_ROOT}/src/github.com/golang/example/outyet'
type: exec
command: go get -d && go run main.go
component: go-cli
- name: stop outyet
actions:
- type: exec
command: kill $(pidof go)
component: go-cli
- name: test outyet
actions:
- workdir: '${CHE_PROJECTS_ROOT}/src/github.com/golang/example/outyet'
type: exec
command: go test
component: go-cli
- name: run current file
actions:
- workdir: '${fileDirname}'
type: exec
command: 'go get -d && go run ${file}'
component: go-cli
- name: Debug current file
actions:
- referenceContent: |
{
"version": "0.2.0",
"configurations": [
{
"name": "Debug current file",
"type": "go",
"request": "launch",
"mode": "auto",
"program": "${fileDirname}",
},
]
}
type: vscode-launch
I started the workspace and i got those logs :
Workspace start logs :
Starting Init Plugin Broker
Cleaning /plugins dir
Unified Che Plugin Broker
Starting VS Code and Theia plugins processing
Downloading VS Code extension for plugin 'ms-vscode/go/0.11.4'
Starting Che plugins and editor processing
List of plugins and editors to install
- ms-vscode/go/0.11.4 - This extension adds rich language support for the Go language
- eclipse/che-machine-exec-plugin/7.3.1 - Che Plug-in with che-machine-exec service to provide creation terminal or tasks for Eclipse CHE workspace containers.
- eclipse/che-theia/7.3.1 - Eclipse Theia
All plugins have been successfully processed
Successfully assigned che/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz to <VM>.<fqdn>
Pulling image "eclipse/che-theia-endpoint-runtime-binary:7.3.1"
Successfully pulled image "eclipse/che-theia-endpoint-runtime-binary:7.3.1"
Created container remote-runtime-injectortsr
Started container remote-runtime-injectortsr
Pulling image "<private registry>/docker/eclipse/che-golang-1.12:7.3.1"
Successfully pulled image "<private registry>/docker/eclipse/che-golang-1.12:7.3.1"
Created container go-cli
Started container go-cli
Pulling image "<private registry>/docker/eclipse/che-machine-exec:7.3.1"
Successfully pulled image "<private registry>/docker/eclipse/che-machine-exec:7.3.1"
Created container che-machine-execfi4
Started container che-machine-execfi4
Pulling image "<private registry>/docker/eclipse/che-theia:7.3.1"
Successfully pulled image "<private registry>/docker/eclipse/che-theia:7.3.1"
Created container theia-ideeul
Started container theia-ideeul
Pulling image "<private registry>/docker/eclipse/che-remote-plugin-go-1.12.9:7.3.1"
Successfully pulled image "<private registry>/docker/eclipse/che-remote-plugin-go-1.12.9:7.3.1"
Created container vscode-godny
Started container vscode-godny
Error: Failed to run the workspace: "Server 'theia' in container 'theia-ideeul' not available."
Theia logs :
kubectl -n che logs -f workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz -c theia-ideeul
chown: /projects: Operation not permitted
root INFO Theia app listening on http://0.0.0.0:3100.
root INFO unzipping the plug-in 'eclipse_che_ports_plugin.theia' to directory: /tmp/theia-unpacked/eclipse_che_ports_plugin.theia
root INFO unzipping the plug-in 'eclipse_che_theia_containers_plugin.theia' to directory: /tmp/theia-unpacked/eclipse_che_theia_containers_plugin.theia
root INFO unzipping the plug-in 'eclipse_che_theia_factory_plugin.theia' to directory: /tmp/theia-unpacked/eclipse_che_theia_factory_plugin.theia
root INFO unzipping the plug-in 'eclipse_che_theia_ssh_plugin.theia' to directory: /tmp/theia-unpacked/eclipse_che_theia_ssh_plugin.theia
root INFO unzipping the plug-in 'eclipse_che_welcome_plugin.theia' to directory: /tmp/theia-unpacked/eclipse_che_welcome_plugin.theia
root INFO unzipping the plug-in 'task_plugin.theia' to directory: /tmp/theia-unpacked/task_plugin.theia
root INFO unzipping the plug-in 'theia_yeoman_plugin.theia' to directory: /tmp/theia-unpacked/theia_yeoman_plugin.theia
root INFO unzipping the VS Code extension 'vscode-git-1.3.0.1.vsix' to directory: /tmp/vscode-unpacked/vscode-git-1.3.0.1.vsix
root ERROR Uncaught Exception: [object Object]
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/theia-unpacked/eclipse_che_ports_plugin.theia
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/theia-unpacked/eclipse_che_theia_containers_plugin.theia
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/theia-unpacked/eclipse_che_theia_factory_plugin.theia
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/theia-unpacked/eclipse_che_theia_ssh_plugin.theia
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/theia-unpacked/eclipse_che_welcome_plugin.theia
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/theia-unpacked/task_plugin.theia
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/theia-unpacked/theia_yeoman_plugin.theia
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/vscode-unpacked/vscode-git-1.3.0.1.vsix
root INFO Resolved "vscode-git-1.3.0.1.vsix" to a VS Code extension "[email protected]" with engines: { vscode: '^1.5.0' }
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /plugins/sidecars
root ERROR Failed to load plugin dependencies from '/plugins/sidecars' path Error: ENOENT: no such file or directory, open '/plugins/sidecars/package.json'
root INFO Deploying backend plugin "@eclipse-che/[email protected]" from "/tmp/theia-unpacked/eclipse_che_ports_plugin.theia/lib/ports-plugin.js"
root INFO Deploying backend plugin "@eclipse-che/[email protected]" from "/tmp/theia-unpacked/eclipse_che_theia_containers_plugin.theia/lib/containers-plugin.js"
root INFO Deploying backend plugin "@eclipse-che/[email protected]" from "/tmp/theia-unpacked/eclipse_che_theia_factory_plugin.theia/lib/factory-plugin.js"
root INFO Deploying backend plugin "@eclipse-che/[email protected]" from "/tmp/theia-unpacked/eclipse_che_theia_ssh_plugin.theia/lib/ssh-plugin-backend.js"
root INFO Deploying backend plugin "@eclipse-che/[email protected]" from "/tmp/theia-unpacked/eclipse_che_welcome_plugin.theia/lib/welcome-plugin.js"
root INFO Deploying backend plugin "[email protected]" from "/tmp/theia-unpacked/task_plugin.theia/lib/task-plugin-backend.js"
root INFO Deploying backend plugin "@theia/[email protected]" from "/tmp/theia-unpacked/theia_yeoman_plugin.theia/lib/theia-yeoman-plugin-backend-plugin.js"
root INFO Deploying backend plugin "[email protected]" from "/tmp/vscode-unpacked/vscode-git-1.3.0.1.vsix/extension/out/main"
kubectl -n che logs -f workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz -c vscode-godny
Starting the deployer with the list of resolvers [ LocalDirectoryPluginDeployerResolver {},
GithubPluginDeployerResolver { unpackedFolder: '/tmp/github-remote' },
HttpPluginDeployerResolver { unpackedFolder: '/tmp/http-remote' },
VsCodePluginDeployerResolver { vscodeExtensionsFolder: '/tmp/vscode-extension-marketplace' } ]
Theia Endpoint 1/pid listening on port 2503
Found the list of default plugins ID on env: undefined
Found the list of plugins ID on env: local-dir:///plugins/sidecars/ms_vscode_go_0_11_4
Found the list of default plugins ID from CLI: undefined
unzipping the VS Code extension 'ms-vscode.go.0.11.4.thkznyafqh.Go-0.11.4.vsix' to directory: /tmp/vscode-unpacked/ms-vscode.go.0.11.4.thkznyafqh.Go-0.11.4.vsix
PluginTheiaDirectoryHandler: accepting plugin with path /tmp/vscode-unpacked/ms-vscode.go.0.11.4.thkznyafqh.Go-0.11.4.vsix
Resolving "ms-vscode.go.0.11.4.thkznyafqh.Go-0.11.4.vsix" as a VS Code extension...
Resolved "ms-vscode.go.0.11.4.thkznyafqh.Go-0.11.4.vsix" to a VS Code extension "[email protected]" with engines: { vscode: '^1.25.0' }
the accepted plugins are []
the acceptedFrontendPlugins plugins are []
the acceptedBackendPlugins plugins are [ PluginDeployerEntryImpl {
originId: 'local-dir:///plugins/sidecars/ms_vscode_go_0_11_4',
pluginId: 'ms-vscode.go.0.11.4.thkznyafqh.Go-0.11.4.vsix',
map: Map { 'package.json' => [Object] },
changes:
[ 'PluginVsCodeFileHandler', 'PluginVsCodeDirectoryHandler' ],
acceptedTypes: [ 1 ],
currentPath:
'/tmp/vscode-unpacked/ms-vscode.go.0.11.4.thkznyafqh.Go-0.11.4.vsix/extension',
initPath:
'/plugins/sidecars/ms_vscode_go_0_11_4/ms-vscode.go.0.11.4.thkznyafqh.Go-0.11.4.vsix',
resolved: true,
resolvedByName: 'LocalDirectoryPluginDeployerResolver' } ]
local path to deploy on remote instance [ '/tmp/vscode-unpacked/ms-vscode.go.0.11.4.thkznyafqh.Go-0.11.4.vsix/extension' ]
Deploying backend plugin "[email protected]" from "/tmp/vscode-unpacked/ms-vscode.go.0.11.4.thkznyafqh.Go-0.11.4.vsix/extension/out/src/goMain"
kubectl -n che get events --sort-by='{.lastTimestamp}' --watch
1s Normal Scheduled pod/mkdir-workspaceffdikzjktx12dbo2 Successfully assigned che/mkdir-workspaceffdikzjktx12dbo2 to <VM-name>017.<fqdn>
1s Normal Pulling pod/mkdir-workspaceffdikzjktx12dbo2 Pulling image "centos:centos7"
1s Normal Pulled pod/mkdir-workspaceffdikzjktx12dbo2 Successfully pulled image "centos:centos7"
1s Normal Created pod/mkdir-workspaceffdikzjktx12dbo2 Created container mkdir-workspaceffdikzjktx12dbo2
1s Normal Started pod/mkdir-workspaceffdikzjktx12dbo2 Started container mkdir-workspaceffdikzjktx12dbo2
1s Normal Scheduled pod/workspaceffdikzjktx12dbo2.che-plugin-broker Successfully assigned che/workspaceffdikzjktx12dbo2.che-plugin-broker to <VM-name>025.<fqdn>
1s Normal Pulling pod/workspaceffdikzjktx12dbo2.che-plugin-broker Pulling image "nadir88/che-init-plugin-broker:nightly"
1s Normal Pulled pod/workspaceffdikzjktx12dbo2.che-plugin-broker Successfully pulled image "nadir88/che-init-plugin-broker:nightly"
1s Normal Created pod/workspaceffdikzjktx12dbo2.che-plugin-broker Created container che-init-plugin-broker-nightly
1s Normal Started pod/workspaceffdikzjktx12dbo2.che-plugin-broker Started container che-init-plugin-broker-nightly
1s Normal Pulling pod/workspaceffdikzjktx12dbo2.che-plugin-broker Pulling image "nadir88/che-unified-plugin-broker:nightly"
1s Normal Pulled pod/workspaceffdikzjktx12dbo2.che-plugin-broker Successfully pulled image "nadir88/che-unified-plugin-broker:nightly"
1s Normal Created pod/workspaceffdikzjktx12dbo2.che-plugin-broker Created container che-unified-plugin-broker-nightly
2s Normal Started pod/workspaceffdikzjktx12dbo2.che-plugin-broker Started container che-unified-plugin-broker-nightly
1s Normal Scheduled pod/mkdir-workspaceffdikzjktx12dbo2 Successfully assigned che/mkdir-workspaceffdikzjktx12dbo2 to <VM-name>017.<fqdn>
1s Normal Pulling pod/mkdir-workspaceffdikzjktx12dbo2 Pulling image "centos:centos7"
1s Normal Pulled pod/mkdir-workspaceffdikzjktx12dbo2 Successfully pulled image "centos:centos7"
1s Normal Created pod/mkdir-workspaceffdikzjktx12dbo2 Created container mkdir-workspaceffdikzjktx12dbo2
1s Normal Started pod/mkdir-workspaceffdikzjktx12dbo2 Started container mkdir-workspaceffdikzjktx12dbo2
2s Normal CREATE ingress/ingress2kks0ak1 Ingress che/ingress2kks0ak1
2s Normal CREATE ingress/ingress2kks0ak1 Ingress che/ingress2kks0ak1
18s Normal CREATE ingress/ingress2kks0ak1 Ingress che/ingress2kks0ak1
2s Normal CREATE ingress/ingress2kks0ak1 Ingress che/ingress2kks0ak1
0s Normal CREATE ingress/ingress2kks0ak1 Ingress che/ingress2kks0ak1
..
1s Normal UPDATE ingress/ingress57febwxx Ingress che/ingress57febwxx
2s Normal UPDATE ingress/ingress6qg1dd74 Ingress che/ingress6qg1dd74
1s Normal ScalingReplicaSet deployment/workspaceffdikzjktx12dbo2.go-cli Scaled up replica set workspaceffdikzjktx12dbo2.go-cli-96d848845 to 1
1s Normal SuccessfulCreate replicaset/workspaceffdikzjktx12dbo2.go-cli-96d848845 Created pod: workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz
1s Normal Scheduled pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Successfully assigned che/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz to <VM-name>025.<fqdn>
2s Normal UPDATE ingress/ingressr5tdm92r Ingress che/ingressr5tdm92r
2s Normal UPDATE ingress/ingressr5tdm92r Ingress che/ingressr5tdm92r
1s Normal Pulling pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Pulling image "eclipse/che-theia-endpoint-runtime-binary:7.3.1"
3s Normal UPDATE ingress/ingressrxa7sfiy Ingress che/ingressrxa7sfiy
3s Normal UPDATE ingress/ingressrxa7sfiy Ingress che/ingressrxa7sfiy
3s Normal UPDATE ingress/ingress57febwxx Ingress che/ingress57febwxx
3s Normal UPDATE ingress/ingress57febwxx Ingress che/ingress57febwxx
1s Normal Pulled pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Successfully pulled image "eclipse/che-theia-endpoint-runtime-binary:7.3.1"
1s Normal Created pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Created container remote-runtime-injectortsr
1s Normal Started pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Started container remote-runtime-injectortsr
1s Normal Pulling pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Pulling image "<private registry>/docker/eclipse/che-golang-1.12:7.3.1"
1s Normal Pulled pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Successfully pulled image "<private registry>/docker/eclipse/che-golang-1.12:7.3.1"
2s Normal Created pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Created container go-cli
1s Normal Started pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Started container go-cli
1s Normal Pulling pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Pulling image "<private registry>/docker/eclipse/che-machine-exec:7.3.1"
1s Normal Pulled pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Successfully pulled image "<private registry>/docker/eclipse/che-machine-exec:7.3.1"
1s Normal Created pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Created container che-machine-execfi4
1s Normal Started pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Started container che-machine-execfi4
1s Normal Pulling pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Pulling image "<private registry>/docker/eclipse/che-theia:7.3.1"
1s Normal Pulled pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Successfully pulled image "<private registry>/docker/eclipse/che-theia:7.3.1"
1s Normal Created pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Created container theia-ideeul
1s Normal Started pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Started container theia-ideeul
1s Normal Pulling pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Pulling image "<private registry>/docker/eclipse/che-remote-plugin-go-1.12.9:7.3.1"
1s Normal Pulled pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Successfully pulled image "<private registry>/docker/eclipse/che-remote-plugin-go-1.12.9:7.3.1"
1s Normal Created pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Created container vscode-godny
1s Normal Started pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Started container vscode-godny
1s Normal DELETE ingress/ingress2kks0ak1 Ingress che/ingress2kks0ak1
1s Normal DELETE ingress/ingress2kks0ak1 Ingress che/ingress2kks0ak1
1s Normal DELETE ingress/ingress2kks0ak1 Ingress che/ingress2kks0ak1
2s Normal DELETE ingress/ingress2kks0ak1 Ingress che/ingress2kks0ak1
18s Normal DELETE ingress/ingress2kks0ak1 Ingress che/ingress2kks0ak1
...
18s Normal DELETE ingress/ingressxre29ylz Ingress che/ingressxre29ylz
1s Normal ScalingReplicaSet deployment/workspaceffdikzjktx12dbo2.go-cli Scaled down replica set workspaceffdikzjktx12dbo2.go-cli-96d848845 to 0
1s Normal Killing pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Stopping container vscode-godny
1s Normal SuccessfulDelete replicaset/workspaceffdikzjktx12dbo2.go-cli-96d848845 Deleted pod: workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz
1s Normal Killing pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Stopping container theia-ideeul
1s Normal Killing pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Stopping container che-machine-execfi4
1s Normal Killing pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Stopping container go-cli
1s Normal Killing pod/workspaceffdikzjktx12dbo2.go-cli-96d848845-fcbqz Stopping container go-cli
NM4
All 49 comments
Hello, this issue is quite important and blocking for us. We would greatly appreciate a quick feedback :)
Plaquevent
on 31 Oct 2019
@NM4 Could you provide more info on how did you deploy Che? Chectl, What are options? Che-operator, What is CheCluster CR?
sleshchenko
on 31 Oct 2019
I assume you have like 5 minute before you get Server 'theia' is not available error.
After all the containers are started, could you manually fetch theia ingress and open it directly in browser? Is this page available? What do you see there?
sleshchenko
on 31 Oct 2019
Exactly ! I have about 5 minutes and I see that all containers are started.
I get two messages the first is 502 Bad Gateway and the second is Cannot GET /(.*)
NM4
on 31 Oct 2019
I deployed che 7.3.1 with Helm only.
I tried che 7.3.0 whith chectl and helm without success too.
NM4
on 31 Oct 2019
@NM4 Seems the issues you face caused by metallb, by Default Che are configured to use Nginx Ingress Controller for exposing public services, like che-server, keyacloak, theia.
Could you provide Ingresses(full Yaml configuration) for Che Server and theia?
sleshchenko
on 31 Oct 2019
The Che Server ingress :
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
field.cattle.io/publicEndpoints: >'[
{"addresses":["","","","","",""],
"port":443,
"protocol":"HTTPS",
"serviceName":"che:che-host",
"ingressName":"che:che-ingress",
"hostname":"**che-che.dns-name**",
"path":"/",
"allNodes":false
}]'
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
creationTimestamp: "2019-10-31T06:54:09Z"
generation: 1
labels:
io.cattle.field/appId: che
name: che-ingress
namespace: che
resourceVersion: "32827623"
selfLink: /apis/extensions/v1beta1/namespaces/che/ingresses/che-ingress
uid: aeac0c5e-5443-489e-9e5a-c08336f3d9dc
spec:
rules:
- host: che-che.<fqdn>
http:
paths:
- backend:
serviceName: che-host
servicePort: 8080
path: /
tls:
- hosts:
- che-che.<fqdn>
secretName: common-ssl-crt
status:
loadBalancer:
ingress:
- hostname: vm1.dns-name
- hostname: vm2.dns-name
- hostname: vm3.dns-name
- hostname: vm4.dns-name
- hostname: vm5.dns-name
- hostname: vm6.dns-name
Theia Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
field.cattle.io/publicEndpoints: >'[
{"addresses":["","","","","",""],
"port":443,
"protocol":"HTTPS",
"serviceName":"che:serverp12xsau8-go-cli",
"ingressName":"che:ingressfvv4q5j0",
"hostname":"serverp12xsau8-go-cli-server-8080.fqdn",
"path":"/(.*)",
"allNodes":false}]'
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/ssl-redirect: "true"
org.eclipse.che.machine.name: go-cli
org.eclipse.che.server.8080/tcp.attributes: '{}'
org.eclipse.che.server.8080/tcp.port: 8080/tcp
org.eclipse.che.server.8080/tcp.protocol: https
creationTimestamp: "2019-10-31T14:46:22Z"
generation: 1
labels:
che.original_name: serverp12xsau8-go-cli-server-8080
che.workspace_id: workspaceffdikzjktx12dbo2
name: ingressfvv4q5j0
namespace: che
resourceVersion: "32989110"
selfLink: /apis/extensions/v1beta1/namespaces/che/ingresses/ingressfvv4q5j0
uid: 25b23eaf-128e-4844-870a-d0da52e726ab
spec:
rules:
- host: serverp12xsau8-go-cli-server-8080.fqdn
http:
paths:
- backend:
serviceName: serverp12xsau8-go-cli
servicePort: server-8080
path: /(.*)
tls:
- hosts:
- serverp12xsau8-go-cli-server-8080.fqdn
secretName: common-ssl-crt
status:
loadBalancer:
ingress:
- hostname: vm.fqdn
- hostname: vm.fqdn
- hostname: vm.fqdn
- hostname: vm.fqdn
- hostname: vm.fqdn
- hostname: vm.fqdn
fqdn are correct DNS entries
NM4
on 31 Oct 2019
The che ingress works fine, I can access che server from other machine.
The dns-nae is a wildcard dns correctly configured on our DNS server and has a routable IP adress that is allocated by metllb.
In the cluster, the ingress controller is Nginx 0.25.1
NM4
on 31 Oct 2019
Are io.cattle.field related fields (labels and annotations) provisioned automatically or you put them there manually? I assume Theia Ingress does not have them.
sleshchenko
on 31 Oct 2019
I used Rancher Catalog, it's Rancher Helm that add these annotations ...
NM4
on 31 Oct 2019
Possible duplicate of https://github.com/eclipse/che/issues/14781
tsmaeder
on 31 Oct 2019
Then maybe "path":"/", vs "path":"/(.*)" in field.cattle.io/publicEndpoints causes an issue.
And it may be caused by nginx.ingress.kubernetes.io/rewrite-target: /$1 that is in Theia ingress but is missing in Che Server Ingress.
@metlos Could you remind why we need rewrite-target for multi-host servers strategy?
sleshchenko
on 31 Oct 2019
@sleshchenko If you prefer, I can redeploy Che with chectl. I can doit quite easily.
NM4
on 31 Oct 2019
Are the two errors in theia logs at startup "acceptable" ?
.
..
root ERROR Uncaught Exception: [object Object]
...
root ERROR Failed to load plugin dependencies from '/plugins/sidecars' path Error: ENOENT: no such file or directory, open '/plugins/sidecars/package.json'
...
NM4
on 31 Oct 2019
@NM4 Not sure, but I believe if it was caused but this - you would get another error but not
I get two messages the first is 502 Bad Gateway and the second is Cannot GET /(.*)
Maybe I may be wrong. @evidolob @mmorhun Do you have some idea?
sleshchenko
on 31 Oct 2019
I redeployed Che with chectl with the same templates that I used with Helm.
This time I get che-remote-plugin-go-1.12.9:7.3.1 in CrashLoopBack.
NM4
on 31 Oct 2019
@NM4 could you please try to start a workspace without remote plugins?
For example (the simplest case):
metadata:
name: wksp-test
components:
- id: eclipse/che-theia/7.3.1
type: cheEditor
apiVersion: 1.0.0
mmorhun
on 1 Nov 2019
@NM4 If Mykola example does not work as well, I would like to ask you to patch helm chart a bit.
Remove rewrite target from workspace ingresses. This line https://github.com/eclipse/che/blob/7.3.1/deploy/kubernetes/helm/che/templates/configmap.yaml should be updated to
CHE_INFRA_KUBERNETES_INGRESS_ANNOTATIONS__JSON: '{"kubernetes.io/ingress.class": "nginx", "kubernetes.io/tls-acme": "true","{{ .Values.global.ingressAnnotationsPrefix }}ingress.kubernetes.io/ssl-redirect": "true","{{ .Values.global.ingressAnnotationsPrefix }}ingress.kubernetes.io/proxy-connect-timeout": "3600","{{ .Values.global.ingressAnnotationsPrefix }}ingress.kubernetes.io/proxy-read-timeout": "3600"}'
Maybe it would be easier for you edit config map on already deployed che ->
kubectl edit configmap che -n che
// remove nginx.ingress.kubernetes.io/rewrite-target": "/$1",
and then to apply changed configuration
kubectl rollout restart deployment/che
@mmorhun here after, che-server logs after workspace creation as described :
2019-11-01 10:29:14,907[nio-8080-exec-5] [INFO ] [o.e.c.a.w.s.WorkspaceManager 347] - Workspace 'workspace6oarr4t6m9owwffs' removed by user 'che'
2019-11-01 10:29:36,450[nio-8080-exec-4] [INFO ] [o.e.c.a.w.s.WorkspaceManager 560] - Workspace 'che/wksp-q2y6' with id 'workspaceauqzupbr66ezsir8' created by user 'che'
2019-11-01 10:30:28,060[nio-8080-exec-3] [DEBUG] [e.EnvVarEnvironmentProvisioner 44] - Provisioning environment variables for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:28,061[nio-8080-exec-3] [DEBUG] [e.EnvVarEnvironmentProvisioner 54] - Environment variables provisioning done for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:28,061[nio-8080-exec-3] [DEBUG] [w.s.s.p.MachineNameProvisioner 36] - Provisioning machine names for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:28,073[nio-8080-exec-3] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 432] - Starting workspace 'che/wksp-test' with id 'workspaceauqzupbr66ezsir8' by user 'che'
2019-11-01 10:30:28,077[aceSharedPool-1] [DEBUG] [.k.w.SidecarToolingProvisioner 69] - Started sidecar tooling provisioning workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:28,082[aceSharedPool-1] [DEBUG] [etesEnvironmentProvisionerImpl 122] - Start provisioning Kubernetes environment for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:28,083[aceSharedPool-1] [DEBUG] [etesEnvironmentProvisionerImpl 125] - Provisioning logs volume for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:28,083[aceSharedPool-1] [DEBUG] [etesEnvironmentProvisionerImpl 130] - Provisioning servers & env vars converters for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:28,083[aceSharedPool-1] [DEBUG] [.c.w.i.k.n.p.CommonPVCStrategy 149] - Provisioning PVC strategy for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:28,084[aceSharedPool-1] [DEBUG] [.c.w.i.k.n.p.CommonPVCStrategy 170] - PVC strategy provisioning done for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:28,085[aceSharedPool-1] [DEBUG] [etesEnvironmentProvisionerImpl 138] - Provisioning environment items for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:28,092[aceSharedPool-1] [DEBUG] [etesEnvironmentProvisionerImpl 151] - Provisioning Kubernetes environment done for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:28,092[aceSharedPool-1] [DEBUG] [.c.w.i.k.w.PluginBrokerManager 119] - Entering plugin brokers deployment chain workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:28,092[aceSharedPool-1] [DEBUG] [c.w.i.k.w.b.ListenBrokerEvents 60] - Subscribing broker events listener for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:28,092[aceSharedPool-1] [DEBUG] [.c.w.i.k.n.p.CommonPVCStrategy 183] - Preparing PVC started for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:28,099[aceSharedPool-1] [DEBUG] [e.c.w.i.k.n.p.PVCSubPathHelper 110] - Preparing PVC `claim-che-workspace` for workspace `workspaceauqzupbr66ezsir8`. Directories to create: [workspaceauqzupbr66ezsir8/che-logs-che-plugin-broker/, workspaceauqzupbr66ezsir8/plugins/]
2019-11-01 10:30:31,271[aceSharedPool-1] [DEBUG] [.c.w.i.k.n.p.CommonPVCStrategy 218] - Preparing PVC done for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:31,272[aceSharedPool-1] [DEBUG] [o.e.c.w.i.k.w.b.DeployBroker 77] - Starting brokers pod for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:31,374[aceSharedPool-1] [DEBUG] [o.e.c.w.i.k.w.b.DeployBroker 105] - Brokers pod is created for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:31,374[aceSharedPool-1] [DEBUG] [e.c.w.i.k.w.b.WaitBrokerResult 63] - Trying to get brokers result for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:41,931[aceSharedPool-1] [DEBUG] [.k.w.SidecarToolingProvisioner 85] - Finished sidecar tooling provisioning workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:41,931[aceSharedPool-1] [DEBUG] [e.EnvVarEnvironmentProvisioner 44] - Provisioning environment variables for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:41,932[aceSharedPool-1] [DEBUG] [e.EnvVarEnvironmentProvisioner 54] - Environment variables provisioning done for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:41,932[aceSharedPool-1] [DEBUG] [w.s.s.p.MachineNameProvisioner 36] - Provisioning machine names for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:41,932[aceSharedPool-1] [DEBUG] [etesEnvironmentProvisionerImpl 122] - Start provisioning Kubernetes environment for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:41,932[aceSharedPool-1] [DEBUG] [etesEnvironmentProvisionerImpl 125] - Provisioning logs volume for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:41,932[aceSharedPool-1] [DEBUG] [etesEnvironmentProvisionerImpl 130] - Provisioning servers & env vars converters for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:41,939[aceSharedPool-1] [DEBUG] [.c.w.i.k.n.p.CommonPVCStrategy 149] - Provisioning PVC strategy for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:41,940[aceSharedPool-1] [DEBUG] [.c.w.i.k.n.p.CommonPVCStrategy 170] - PVC strategy provisioning done for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:41,941[aceSharedPool-1] [DEBUG] [etesEnvironmentProvisionerImpl 138] - Provisioning environment items for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:41,944[aceSharedPool-1] [DEBUG] [etesEnvironmentProvisionerImpl 151] - Provisioning Kubernetes environment done for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:41,944[aceSharedPool-1] [DEBUG] [.i.k.KubernetesInternalRuntime 197] - Provisioning of workspace 'workspaceauqzupbr66ezsir8' completed.
2019-11-01 10:30:41,944[aceSharedPool-1] [DEBUG] [.c.w.i.k.n.p.CommonPVCStrategy 183] - Preparing PVC started for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:41,950[aceSharedPool-1] [DEBUG] [e.c.w.i.k.n.p.PVCSubPathHelper 110] - Preparing PVC `claim-che-workspace` for workspace `workspaceauqzupbr66ezsir8`. Directories to create: [workspaceauqzupbr66ezsir8/che-logs-che-workspace-pod/, workspaceauqzupbr66ezsir8/plugins/, workspaceauqzupbr66ezsir8/projects/]
2019-11-01 10:30:45,762[aceSharedPool-1] [DEBUG] [.c.w.i.k.n.p.CommonPVCStrategy 218] - Preparing PVC done for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:30:45,925[aceSharedPool-1] [DEBUG] [.i.k.KubernetesInternalRuntime 767] - Ingresses created for workspace 'workspaceauqzupbr66ezsir8'. Wait them to be ready.
2019-11-01 10:31:06,775[aceSharedPool-1] [DEBUG] [.i.k.KubernetesInternalRuntime 785] - Ingresses creation for workspace 'workspaceauqzupbr66ezsir8' done.
2019-11-01 10:31:06,832[aceSharedPool-1] [DEBUG] [.i.k.KubernetesInternalRuntime 667] - Begin pods creation for workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:31:06,951[aceSharedPool-1] [DEBUG] [.i.k.KubernetesInternalRuntime 671] - Creating pod 'workspaceauqzupbr66ezsir8.che-workspace-pod' in workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:31:06,951[aceSharedPool-1] [DEBUG] [.i.k.KubernetesInternalRuntime 698] - Creating machine 'che-machine-execs2f' in workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:31:06,956[aceSharedPool-1] [DEBUG] [.i.k.KubernetesInternalRuntime 698] - Creating machine 'theia-idehki' in workspace 'workspaceauqzupbr66ezsir8'2019-11-01 10:31:06,964[aceSharedPool-1] [DEBUG] [.i.k.KubernetesInternalRuntime 684] - Pods creation finished in workspace 'workspaceauqzupbr66ezsir8'2019-11-01 10:31:07,098[aceSharedPool-1] [DEBUG] [.i.k.KubernetesInternalRuntime 329] - Waiting to start machines of workspace 'workspaceauqzupbr66ezsir8'2019-11-01 10:31:18,259[ineSharedPool-4] [DEBUG] [.i.k.KubernetesInternalRuntime 395] - Performing servers check for machine 'che-machine-execs2f' in workspace 'workspaceauqzupbr66ezsir8'2019-11-01 10:31:18,259[ineSharedPool-5] [DEBUG] [.i.k.KubernetesInternalRuntime 395] - Performing servers check for machine 'theia-idehki' in workspace 'workspaceauqzupbr66ezsir8'2019-11-01 10:31:18,260[ineSharedPool-4] [DEBUG] [.i.k.KubernetesInternalRuntime 404] - Servers checks done for machine 'che-machine-execs2f' in workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:34:18,793[aceSharedPool-1] [WARN ] [.i.k.KubernetesInternalRuntime 241] - Failed to start Kubernetes runtime of workspace workspaceauqzupbr66ezsir8. Cause: Server 'theia' in container 'theia-idehki' not available.
2019-11-01 10:34:18,795[ServersChecker] [DEBUG] [.i.k.KubernetesInternalRuntime 404] - Servers checks done for machine 'theia-idehki' in workspace 'workspaceauqzupbr66ezsir8'
2019-11-01 10:34:19,348[aceSharedPool-1] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 854] - Workspace 'che:wksp-test' with id 'workspaceauqzupbr66ezsir8' start failed
@NM4 thanks for testing. From the logs, seems the remote plugins mechanism is not the root cause of the issue and the problem is, probably, in endpoints accessibility.
Once I had some strange behaviour with routes and the cause was wrong userland proxy option. @sleshchenko is it possibly a case in this situation?
mmorhun
on 1 Nov 2019
@NM4 Please take a look https://github.com/eclipse/che/issues/15043#issuecomment-548711444
sleshchenko
on 1 Nov 2019
@sleshchenko
I do what what you described, here after my cm :
apiVersion: v1
data:
CHE_API: https://che-che.fqdn/api
CHE_CORS_ALLOW__CREDENTIALS: "false"
CHE_CORS_ALLOWED__ORIGINS: '*'
CHE_CORS_ENABLED: "true"
CHE_DEBUG_SERVER: "true"
CHE_HOST: che-che.fqdn
CHE_INFRA_KUBERNETES_BOOTSTRAPPER_BINARY__URL: https://che-che.fqdn/agent-binaries/linux_amd64/bootstrapper/bootstrapper
CHE_INFRA_KUBERNETES_INGRESS_ANNOTATIONS__JSON: '{"kubernetes.io/ingress.class":
"nginx", "nginx.ingress.kubernetes.io/ssl-redirect": "true","nginx.ingress.kubernetes.io/proxy-connect-timeout":
"3600","nginx.ingress.kubernetes.io/proxy-read-timeout": "3600"}'
CHE_INFRA_KUBERNETES_INGRESS_DOMAIN: fqdn
CHE_INFRA_KUBERNETES_INGRESS_PATH__TRANSFORM: '%s(.*)'
CHE_INFRA_KUBERNETES_MACHINE__START__TIMEOUT__MIN: "5"
CHE_INFRA_KUBERNETES_MASTER__URL: ""
CHE_INFRA_KUBERNETES_NAMESPACE: che
CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_FS__GROUP: "1724"
CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_RUN__AS__USER: "1724"
CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS: "true"
CHE_INFRA_KUBERNETES_PVC_QUANTITY: 1Gi
CHE_INFRA_KUBERNETES_PVC_STORAGE__CLASS__NAME: rook-ceph-block
CHE_INFRA_KUBERNETES_PVC_STRATEGY: common
CHE_INFRA_KUBERNETES_SERVER__STRATEGY: multi-host
CHE_INFRA_KUBERNETES_SERVICE__ACCOUNT__NAME: che-workspace
CHE_INFRA_KUBERNETES_TLS__ENABLED: "true"
CHE_INFRA_KUBERNETES_TLS__SECRET: common-ssl-crt
CHE_INFRA_KUBERNETES_TRUST__CERTS: "false"
CHE_INFRA_KUBERNETES_WORKSPACE__START__TIMEOUT__MIN: "15"
CHE_INFRASTRUCTURE_ACTIVE: kubernetes
CHE_LIMITS_WORKSPACE_IDLE_TIMEOUT: "1800000"
CHE_LOCAL_CONF_DIR: /etc/conf
CHE_LOG_LEVEL: INFO
CHE_LOGGER_CONFIG: org.eclipse.che.workspace.infrastructure.kubernetes=DEBUG,org.eclipse.che.api.workspace.server=DEBUG
CHE_LOGS_APPENDERS_IMPL: plaintext
CHE_LOGS_DIR: /data/logs
CHE_METRICS_ENABLED: "false"
CHE_MULTIUSER: "false"
CHE_OAUTH_GITHUB_CLIENTID: ""
CHE_OAUTH_GITHUB_CLIENTSECRET: ""
CHE_PORT: "8080"
CHE_SERVER_SECURE__EXPOSER_JWTPROXY_IMAGE: eclipse/che-jwtproxy:dbd0578
CHE_TRACING_ENABLED: "false"
CHE_WEBSOCKET_ENDPOINT: wss://che-che.fqdn/api/websocket
CHE_WORKSPACE_AUTO_START: "false"
CHE_WORKSPACE_DEVFILE__REGISTRY__URL: https://devfile-registry-che.fqdn
CHE_WORKSPACE_HTTP__PROXY: ""
CHE_WORKSPACE_HTTPS__PROXY: ""
CHE_WORKSPACE_JAVA__OPTIONS: -Xmx2000m
CHE_WORKSPACE_MAVEN__OPTIONS: -Xmx20000m
CHE_WORKSPACE_NO__PROXY: ""
CHE_WORKSPACE_PLUGIN__BROKER_INIT_IMAGE: nadir88/che-init-plugin-broker:nightly
CHE_WORKSPACE_PLUGIN__BROKER_UNIFIED_IMAGE: nadir88/che-unified-plugin-broker:nightly
CHE_WORKSPACE_PLUGIN__REGISTRY__URL: https://plugin-registry-che.fqdn/v3
CHE_WSAGENT_CORS_ALLOW__CREDENTIALS: "true"
CHE_WSAGENT_CORS_ALLOWED__ORIGINS: "NULL"
CHE_WSAGENT_CORS_ENABLED: "true"
JAEGER_ENDPOINT: http://jaeger-collector:14268/api/traces
JAEGER_REPORTER_MAX_QUEUE_SIZE: "10000"
JAEGER_SAMPLER_MANAGER_HOST_PORT: jaeger:5778
JAEGER_SAMPLER_PARAM: "1"
JAEGER_SAMPLER_TYPE: const
JAEGER_SERVICE_NAME: che-server
JAVA_OPTS: -XX:MaxRAMFraction=2 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20
-XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+UnlockExperimentalVMOptions
-XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -Xms20m
But when I start a new fresh workspace I get ingresses with /(.*)
kubectl -n che describe ingress ingressge3kpe3z
Name: ingressge3kpe3z
Namespace: che
Address: vm1.fqdn,vm2.fqdn,vm3.fqdn,vm4.fqdn,vm5.fqdn,vm6.fqdn
Default backend: default-http-backend:80 (<none>)
TLS:
common-ssl-crt terminates servercsr2wp56-theia-ide05w-server-13131.fqdn
Rules:
Host Path Backends
---- ---- --------
servercsr2wp56-theia-ide05w-server-13131.fqdn
/(.*) servercsr2wp56-theia-ide05w:server-13131 (10.42.8.169:13131)
Annotations:
field.cattle.io/publicEndpoints: [{"addresses":["","","","","",""],"port":443,"protocol":"HTTPS","serviceName":"che:servercsr2wp56-theia-ide05w","ingressName":"che:ingressge3kpe3z","hostname":"servercsr2wp56-theia-ide05w-server-13131.fqdn","path":"/(.*)","allNodes":false}]
kubernetes.io/tls-acme: true
nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
org.eclipse.che.server.theia-redirect-1.attributes: {"internal":"false","discoverable":"false"}
org.eclipse.che.server.theia-redirect-1.port: 13131/tcp
org.eclipse.che.server.theia-redirect-1.protocol: https
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/proxy-connect-timeout: 3600
nginx.ingress.kubernetes.io/ssl-redirect: true
org.eclipse.che.machine.name: theia-ide05w
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 116s nginx-ingress-controller Ingress che/ingressge3kpe3z
Normal CREATE 93s nginx-ingress-controller Ingress che/ingressge3kpe3z
Normal CREATE 93s nginx-ingress-controller Ingress che/ingressge3kpe3z
Normal CREATE 93s nginx-ingress-controller Ingress che/ingressge3kpe3z
Normal CREATE 92s nginx-ingress-controller Ingress che/ingressge3kpe3z
Normal CREATE 91s nginx-ingress-controller Ingress che/ingressge3kpe3z
Normal UPDATE 75s (x2 over 75s) nginx-ingress-controller Ingress che/ingressge3kpe3z
Normal UPDATE 52s (x2 over 52s) nginx-ingress-controller Ingress che/ingressge3kpe3z
Normal UPDATE 52s (x2 over 52s) nginx-ingress-controller Ingress che/ingressge3kpe3z
Normal UPDATE 52s (x2 over 52s) nginx-ingress-controller Ingress che/ingressge3kpe3z
Normal UPDATE 51s (x2 over 52s) nginx-ingress-controller Ingress che/ingressge3kpe3z
Normal UPDATE 50s (x2 over 50s) nginx-ingress-controller Ingress che/ingressge3kpe3z
@sleshchenko
In the cm there is anothere parameter about ingress :
CHE_INFRA_KUBERNETES_INGRESS_PATH__TRANSFORM: '%s(.*)'
should I modify it too ?
NM4
on 1 Nov 2019
@NM4 Right! I just took a look at it!!!
I'm not sure if it helps, @metlos knows better.
But I assume that value like / may help.
P.S. Maybe %s is the right value.
sleshchenko
on 1 Nov 2019
@sleshchenko I get the same error with %s
NM4
on 1 Nov 2019
I created a new workspace with Eclipse-ide as cheEditor, I don't get anything displayed but the created ingress is OK. and the workspace still runing.
NM4
on 1 Nov 2019
I restarted Che server with cm containing : (default values)
CHE_INFRA_KUBERNETES_INGRESS_ANNOTATIONS__JSON : {"kubernetes.io/ingress.class": "nginx", "kubernetes.io/tls-acme": "true", "nginx.ingress.kubernetes.io/rewrite-target": "/$1","nginx.ingress.kubernetes.io/ssl-redirect": "true","nginx.ingress.kubernetes.io/proxy-connect-timeout": "3600","nginx.ingress.kubernetes.io/proxy-read-timeout": "3600"}
CHE_INFRA_KUBERNETES_INGRESS_PATH__TRANSFORM : %s(.*)
I created a workspace with this workspace devfile :
metadata:
name: wksp-test
components:
- id: ws-skeleton/eclipseide/latest
type: cheEditor
apiVersion: 1.0.0
Then I get "Import Projects from FileSystem or Archive" window displayed !
NM4
on 1 Nov 2019
I restarted Che server with cm containing : (default values)
o_O Do I understand correctly that you returned back to the start point and now it works at least with Theia Edit only?
sleshchenko
on 1 Nov 2019
@sleshchenko
Yes when I create a workspace without remote plugin and Eclipse-ide as editor it works.
NM4
on 4 Nov 2019
Hello,
Any update please on this issue ?
As summary :
I deployed che with chectl on k8s platform bootstrapped with Rancher RKE and the ingress-controller is exposed with metallb loadbalancer.
When I create a workspace with this Devfiles :
metadata:
name: wksp-test
components:
- id: ws-skeleton/eclipseide/latest
type: cheEditor
apiVersion: 1.0.0
I get the Eclipse Ide interface.
But with
metadata:
name: wksp-test
components:
- id: eclipse/che-theia/7.3.1
type: cheEditor
apiVersion: 1.0.0
I get
Error: Failed to run the workspace: "Server 'theia' in container 'theia-ideaff' not available."
with ep created :
kubectl -n che get ep
NAME ENDPOINTS AGE
che-host 10.42.8.3:8080,10.42.8.3:8087 49m
devfile-registry 10.42.8.2:8080 49m
plugin-registry 10.42.6.169:8080 49m
server0blqlr97-theia-idek1f 10.42.8.13:13132,10.42.8.13:13131,10.42.8.13:3130 + 2 more... 92s
servertog353f2-che-machine-execcws 10.42.8.13:4444 92s
and che-server logs :
2019-11-05 13:31:22,704[aceSharedPool-9] [DEBUG] [.i.k.KubernetesInternalRuntime 678] - Creating deployment 'workspaceufop0p3tkbh8siy5.maven' in workspace 'workspaceufop0p3tkbh8siy5'
2019-11-05 13:31:22,705[aceSharedPool-9] [DEBUG] [.i.k.KubernetesInternalRuntime 698] - Creating machine 'maven' in workspace 'workspaceufop0p3tkbh8siy5'
2019-11-05 13:31:22,706[aceSharedPool-9] [DEBUG] [.i.k.KubernetesInternalRuntime 698] - Creating machine 'che-machine-execcws' in workspace 'workspaceufop0p3tkbh8siy5'
2019-11-05 13:31:22,708[aceSharedPool-9] [DEBUG] [.i.k.KubernetesInternalRuntime 698] - Creating machine 'theia-idek1f' in workspace 'workspaceufop0p3tkbh8siy5'
2019-11-05 13:31:22,710[aceSharedPool-9] [DEBUG] [.i.k.KubernetesInternalRuntime 698] - Creating machine 'vscode-javaldv' in workspace 'workspaceufop0p3tkbh8siy5'
2019-11-05 13:31:22,711[aceSharedPool-9] [DEBUG] [.i.k.KubernetesInternalRuntime 684] - Pods creation finished in workspace 'workspaceufop0p3tkbh8siy5'
2019-11-05 13:31:22,962[aceSharedPool-9] [DEBUG] [.i.k.KubernetesInternalRuntime 329] - Waiting to start machines of workspace 'workspaceufop0p3tkbh8siy5'
2019-11-05 13:31:31,371[neSharedPool-13] [DEBUG] [.i.k.KubernetesInternalRuntime 395] - Performing servers check for machine 'che-machine-execcws' in workspace 'workspaceufop0p3tkbh8siy5'
2019-11-05 13:31:31,371[neSharedPool-15] [DEBUG] [.i.k.KubernetesInternalRuntime 395] - Performing servers check for machine 'theia-idek1f' in workspace 'workspaceufop0p3tkbh8siy5'
2019-11-05 13:31:31,373[neSharedPool-13] [DEBUG] [.i.k.KubernetesInternalRuntime 404] - Servers checks done for machine 'che-machine-execcws' in workspace 'workspaceufop0p3tkbh8siy5'
2019-11-05 13:31:31,377[neSharedPool-14] [DEBUG] [.i.k.KubernetesInternalRuntime 395] - Performing servers check for machine 'vscode-javaldv' in workspace 'workspaceufop0p3tkbh8siy5'
2019-11-05 13:31:31,377[neSharedPool-16] [DEBUG] [.i.k.KubernetesInternalRuntime 395] - Performing servers check for machine 'maven' in workspace 'workspaceufop0p3tkbh8siy5'
2019-11-05 13:31:31,377[neSharedPool-16] [DEBUG] [.i.k.KubernetesInternalRuntime 404] - Servers checks done for machine 'maven' in workspace 'workspaceufop0p3tkbh8siy5'
2019-11-05 13:31:31,378[neSharedPool-14] [DEBUG] [.i.k.KubernetesInternalRuntime 404] - Servers checks done for machine 'vscode-javaldv' in workspace 'workspaceufop0p3tkbh8siy5'
2019-11-05 13:34:31,922[aceSharedPool-9] [WARN ] [.i.k.KubernetesInternalRuntime 241] - Failed to start Kubernetes runtime of workspace workspaceufop0p3tkbh8siy5. Cause: Server 'theia' in container 'theia-idek1f' not available.
2019-11-05 13:34:31,924[ServersChecker] [DEBUG] [.i.k.KubernetesInternalRuntime 404] - Servers checks done for machine 'theia-idek1f' in workspace 'workspaceufop0p3tkbh8siy5'
2019-11-05 13:34:32,544[aceSharedPool-9] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 854] - Workspace 'che:wksp-v9h3' with id 'workspaceufop0p3tkbh8siy5' start failed
NM4
on 5 Nov 2019
Hello
any hints about @NM4 issues ?
EDIT : do you know a way to get theia logs ?
TristanFAURE
on 6 Nov 2019
The last idea I have - permissions issue in the mounted volumes.
From the shared Theia Logs:
chown: /projects: Operation not permitted
...
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /plugins/sidecars
root ERROR Failed to load plugin dependencies from '/plugins/sidecars' path Error: ENOENT: no such file or directory, open '/plugins/sidecars/package.json'
@NM4 Are you able to share logs of Theia without plugins?
Are you able to open terminal in Theia container and try to create any file into /plugins, /projects folders? What are permissions for these directories?
sleshchenko
on 6 Nov 2019
@mmorhun @evidolob Can non-writable /projects, /plugins prevent Theia from starting? If yes, can we change it, like theia should show an error message but start without any plugins loaded?
sleshchenko
on 6 Nov 2019
@sleshchenko thanks for replying.
Yes I'm able to create files in /projects and /plugins :
kubectl -n che exec -ti workspacextzj78sc4rl3hbbu.go-cli-b9c8c599c-llkfr -c theia-ide768 /bin/bash
bash-4.4$ cd /
bash-4.4$ ls -la
total 84
drwxr-xr-x 1 0 root 4096 Nov 6 09:55 .
drwxr-xr-x 1 0 root 4096 Nov 6 09:55 ..
-rwxr-xr-x 1 0 root 0 Nov 6 09:55 .dockerenv
drwxr-xr-x 1 0 root 4096 Oct 30 14:33 bin
drwxrwxr-x 1 0 root 4096 Oct 30 14:32 default-theia-plugins
drwxr-xr-x 5 0 root 360 Nov 6 09:55 dev
-rwxrwxr-x 1 0 root 2595 Aug 5 13:25 entrypoint.sh
drwxr-xr-x 1 0 root 4096 Nov 6 09:55 etc
drwxr-xr-x 1 0 root 4096 Oct 30 14:33 home
drwxr-xr-x 1 0 root 4096 May 9 20:49 lib
drwxr-xr-x 5 0 root 4096 May 9 20:49 media
drwxr-xr-x 2 0 root 4096 May 9 20:49 mnt
drwxrwxr-x 2 0 root 4096 Oct 30 14:33 node_modules
drwxr-xr-x 1 0 root 4096 May 29 17:18 opt
drwxrwsrwx 3 root 1724 22 Nov 6 09:54 plugins
dr-xr-xr-x 410 0 root 0 Nov 6 09:55 proc
drwxrwxrwx 2 root root 18 Nov 6 09:52 projects
drwx------ 1 0 root 4096 May 29 16:25 root
drwxr-xr-x 1 0 root 4096 Nov 6 09:55 run
drwxr-xr-x 2 0 root 4096 May 9 20:49 sbin
drwxr-xr-x 2 0 root 4096 May 9 20:49 srv
dr-xr-xr-x 13 0 root 0 Nov 6 09:55 sys
drwxrwxrwt 1 0 root 4096 Nov 6 09:55 tmp
drwxr-xr-x 1 0 root 4096 Oct 30 14:33 usr
drwxr-xr-x 1 0 root 4096 Oct 30 14:33 var
drwxrwsrwx 2 root 1724 6 Nov 5 18:07 workspace_logs
bash-4.4$ cd projects
bash-4.4$ touch hello
bash-4.4$ ls -la
total 4
drwxrwxrwx 2 root root 31 Nov 6 09:56 .
drwxr-xr-x 1 0 root 4096 Nov 6 09:55 ..
-rw-r--r-- 1 root root 0 Nov 6 09:56 hello
-rw-rw-r-- 1 root 1724 0 Nov 6 09:52 toto
bash-4.4$ cd /
bash-4.4$ cd plugins
bash-4.4$ touch hello
bash-4.4$ ls -la
total 4
drwxrwsrwx 3 root 1724 35 Nov 6 09:56 .
drwxr-xr-x 1 0 root 4096 Nov 6 09:55 ..
-rw-r--r-- 1 root 1724 0 Nov 6 09:56 hello
drwxrwsr-x 3 root 1724 33 Nov 6 09:54 sidecars
bash-4.4$ cd sidecars/
bash-4.4$ ls -la
total 0
drwxrwsr-x 3 root 1724 33 Nov 6 09:54 .
drwxrwsrwx 3 root 1724 35 Nov 6 09:56 ..
drwxrwsr-x 2 root 1724 59 Nov 6 09:54 ms_vscode_go_latest
bash-4.4$ pwd
/plugins/sidecars
bash-4.4$ ls -la
total 0
drwxrwsr-x 3 root 1724 33 Nov 6 09:54 .
drwxrwsrwx 3 root 1724 35 Nov 6 09:56 ..
drwxrwsr-x 2 root 1724 59 Nov 6 09:54 ms_vscode_go_latest
NM4
on 6 Nov 2019
Is there a way to launch Theia workspace in debug mod ?
NM4
on 6 Nov 2019
With this devfile :
metadata:
name: wksp-test
components:
- id: eclipse/che-theia/7.3.1
type: cheEditor
apiVersion: 1.0.0
I have Theia logs bellow :
root INFO Theia app listening on http://0.0.0.0:3100.
root INFO unzipping the plug-in 'eclipse_che_ports_plugin.theia' to directory: /tmp/theia-unpacked/eclipse_che_ports_plugin.theia
root INFO unzipping the plug-in 'eclipse_che_theia_containers_plugin.theia' to directory: /tmp/theia-unpacked/eclipse_che_theia_containers_plugin.theia
root INFO unzipping the plug-in 'eclipse_che_theia_factory_plugin.theia' to directory: /tmp/theia-unpacked/eclipse_che_theia_factory_plugin.theia
root INFO unzipping the plug-in 'eclipse_che_theia_ssh_plugin.theia' to directory: /tmp/theia-unpacked/eclipse_che_theia_ssh_plugin.theia
root INFO unzipping the plug-in 'eclipse_che_welcome_plugin.theia' to directory: /tmp/theia-unpacked/eclipse_che_welcome_plugin.theia
root INFO unzipping the plug-in 'task_plugin.theia' to directory: /tmp/theia-unpacked/task_plugin.theia
root INFO unzipping the plug-in 'theia_yeoman_plugin.theia' to directory: /tmp/theia-unpacked/theia_yeoman_plugin.theia
root INFO unzipping the VS Code extension 'vscode-git-1.3.0.1.vsix' to directory: /tmp/vscode-unpacked/vscode-git-1.3.0.1.vsix
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/theia-unpacked/eclipse_che_ports_plugin.theia
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/theia-unpacked/eclipse_che_theia_containers_plugin.theia
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/theia-unpacked/eclipse_che_theia_factory_plugin.theia
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/theia-unpacked/eclipse_che_theia_ssh_plugin.theia
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/theia-unpacked/eclipse_che_welcome_plugin.theia
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/theia-unpacked/task_plugin.theia
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/theia-unpacked/theia_yeoman_plugin.theia
root INFO PluginTheiaDirectoryHandler: accepting plugin with path /tmp/vscode-unpacked/vscode-git-1.3.0.1.vsix
root INFO Resolved "vscode-git-1.3.0.1.vsix" to a VS Code extension "[email protected]" with engines: { vscode: '^1.5.0' }
root INFO Deploying backend plugin "@eclipse-che/[email protected]" from "/tmp/theia-unpacked/eclipse_che_ports_plugin.theia/lib/ports-plugin.js"
root INFO Deploying backend plugin "@eclipse-che/[email protected]" from "/tmp/theia-unpacked/eclipse_che_theia_containers_plugin.theia/lib/containers-plugin.js"
root INFO Deploying backend plugin "@eclipse-che/[email protected]" from "/tmp/theia-unpacked/eclipse_che_theia_factory_plugin.theia/lib/factory-plugin.js"
root INFO Deploying backend plugin "@eclipse-che/[email protected]" from "/tmp/theia-unpacked/eclipse_che_theia_ssh_plugin.theia/lib/ssh-plugin-backend.js"
root INFO Deploying backend plugin "@eclipse-che/[email protected]" from "/tmp/theia-unpacked/eclipse_che_welcome_plugin.theia/lib/welcome-plugin.js"
root INFO Deploying backend plugin "[email protected]" from "/tmp/theia-unpacked/task_plugin.theia/lib/task-plugin-backend.js"
root INFO Deploying backend plugin "@theia/[email protected]" from "/tmp/theia-unpacked/theia_yeoman_plugin.theia/lib/theia-yeoman-plugin-backend-plugin.js"
root INFO Deploying backend plugin "[email protected]" from "/tmp/vscode-unpacked/vscode-git-1.3.0.1.vsix/extension/out/main"
NM4
on 6 Nov 2019
@NM4 There are no any errors in Thea logs you shared, but you still get 502 when you go to the Theia's URL?
sleshchenko
on 6 Nov 2019
Here what I get for each created ingress :
https://server8qz07s6b-theia-idev8p-server-13131.fqdn/(.*) : 502 Bad Gateway openresty/1.15.8.1
https://serverhrb66bv9-che-machine-exec9wo-server-4444.fqdn/(.*) : 404 page not found
https://server8qz07s6b-theia-idev8p-server-3130.fqdn/(.*) : 502 Bad Gateway openresty/1.15.8.1
https://server8qz07s6b-theia-idev8p-server-13133.fqdn/(.*) : 502 Bad Gateway openresty/1.15.8.1
https://server8qz07s6b-theia-idev8p-server-13132.fqdn/(.*) : 502 Bad Gateway openresty/1.15.8.1
https://server8qz07s6b-theia-idev8p-server-3100.fqdn/(.*) : Cannot GET /(.*)
And here che-server UI logs :
Starting Init Plugin Broker
Cleaning /plugins dir
Unified Che Plugin Broker
List of plugins and editors to install
- eclipse/che-machine-exec-plugin/7.3.1 - Che Plug-in with che-machine-exec service to provide creation terminal or tasks for Eclipse CHE workspace containers.
- eclipse/che-theia/7.3.1 - Eclipse Theia
Starting Che plugins and editor processing
All plugins have been successfully processed
Starting VS Code and Theia plugins processing
Error: Failed to run the workspace: "Server 'theia' in container 'theia-idev8p' not available."
@NM4
https://server8qz07s6b-theia-idev8p-server-13131.fqdn/(.*) : 502 Bad Gateway openresty/1.15.8.1
Could you explain (.*) fragment? It is used by Che? Are you able to access Theia if you open URL without such fragment?
sleshchenko
on 6 Nov 2019
Yes this is the ingress created by Che (hostname + path) :
kubectl -n che describe ing ingressh3rkv3r5
Name: ingressh3rkv3r5
Namespace: che
Address: vm.fqdn,dvodvoks018.fqdn,vm.fqdn,vm.fqdn,vm.fqdn,vm.fqdn
Default backend: default-http-backend:80 (<none>)
TLS:
che-tls terminates serveranxg9fws-theia-ide5w4-server-13131.fqdn
Rules:
Host Path Backends
---- ---- --------
serveranxg9fws-theia-ide5w4-server-13131.fqdn
/(.*) serveranxg9fws-theia-ide5w4:server-13131 (10.42.6.248:13131)
Annotations:
nginx.ingress.kubernetes.io/proxy-connect-timeout: 3600
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/ssl-redirect: true
org.eclipse.che.server.theia-redirect-1.attributes: {"internal":"false","discoverable":"false"}
org.eclipse.che.server.theia-redirect-1.port: 13131/tcp
org.eclipse.che.server.theia-redirect-1.protocol: https
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: true
org.eclipse.che.machine.name: theia-ide5w4
field.cattle.io/publicEndpoints: [{"addresses":["","","","","",""],"port":443,"protocol":"HTTPS","serviceName":"che:serveranxg9fws-theia-ide5w4","ingressName":"che:ingressh3rkv3r5","hostname":"serveranxg9fws-theia-ide5w4-server-13131.fqdn","path":"/(.*)","allNodes":false}]
nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 4m14s nginx-ingress-controller Ingress che/ingressh3rkv3r5
Normal UPDATE 3m18s (x2 over 3m18s) nginx-ingress-controller Ingress che/ingressh3rkv3r5
Normal CREATE 3m14s nginx-ingress-controller Ingress che/ingressh3rkv3r5
Normal CREATE 3m14s nginx-ingress-controller Ingress che/ingressh3rkv3r5
Normal CREATE 3m14s nginx-ingress-controller Ingress che/ingressh3rkv3r5
Normal CREATE 3m14s nginx-ingress-controller Ingress che/ingressh3rkv3r5
Normal CREATE 3m13s nginx-ingress-controller Ingress che/ingressh3rkv3r5
Normal UPDATE 2m19s (x2 over 2m19s) nginx-ingress-controller Ingress che/ingressh3rkv3r5
Normal UPDATE 2m18s (x2 over 2m18s) nginx-ingress-controller Ingress che/ingressh3rkv3r5
Normal UPDATE 2m18s (x2 over 2m18s) nginx-ingress-controller Ingress che/ingressh3rkv3r5
Normal UPDATE 2m18s (x2 over 2m18s) nginx-ingress-controller Ingress che/ingressh3rkv3r5
Normal UPDATE 2m17s (x2 over 2m17s) nginx-ingress-controller Ingress che/ingressh3rkv3r5
Without (.*) I get the same message 502 Bad Gateway
NM4
on 6 Nov 2019
@NM4 can you check the version of the nginx-ingress-controller? Something along the lines of:
kubectl get deployment nginx-ingress-controller -n kube-system -o wide
There is a lengthy description of the ingress annotation config:
https://github.com/eclipse/che/blob/3de4e7ffac7e6f9865dc3b3e1b47ff16c2d25544/assembly/assembly-wsmaster-war/src/main/webapp/WEB-INF/classes/che/che.properties#L354
The two env vars corresponding to the config props mentioned in the che.properties are CHE_INFRA_KUBERNETES_INGRESS_ANNOTATIONS__JSON and CHE_INFRA_KUBERNETES_INGERSS_PATH__TRANSFORM.
And yes, there is a bug in the descriptions, where the property in the comments is called che.infra.kubernetes.ingress.path.rewrite_transform instead of the actual che.infra.kubernetes.ingress.path_transform.
metlos
on 7 Nov 2019
@metlos
I have nginx ingress controller version 0.25.1
Here after what I had in che configMap :
CHE_INFRA_KUBERNETES_INGRESS_ANNOTATIONS__JSON: '{"kubernetes.io/ingress.class":
"nginx", "nginx.ingress.kubernetes.io/rewrite-target": "/$1","nginx.ingress.kubernetes.io/ssl-redirect":
"false","nginx.ingress.kubernetes.io/proxy-connect-timeout": "3600","nginx.ingress.kubernetes.io/proxy-read-timeout":
"3600"}'
CHE_INFRA_KUBERNETES_INGRESS_PATH__TRANSFORM: '%s(.*)'
I redeployed Che without HTTPS/ SSL and now, when I start an empty workspace with Theia as editor. I get all pods in running state and stay in this state without any error, But Theia UI is not displayed.
We analyzed the web page with Firefox developer tools, and we have seen that there are links to redhat.com for exemple :
https://static.developers.redhat.com/che/theia_artifacts/che.103ff3fa440e96277bc7.js
Or we are in an offline environment.
Is there a way to get Theia working on Air-gap environment ?
NM4
on 7 Nov 2019
^ cc @amisevsk
sleshchenko
on 8 Nov 2019
Hello @sleshchenko @amisevsk
I confirm that I have the same behavior with Minikube :
I Installed eclipse che with chectl (7.4.0)
I launch a workspace with Theia as ide
My workspace become up and running so I can use it.
I switch to offline mode ( disconnect my Mac from the network)
I refresh my Che workspace and the I get nothing displayed and when I dig HTML/JS I get the same error as before ( URL to static.developers.redhat.com )
Thank you for your help.
NM4
on 12 Nov 2019
Hm, I have no idea why https://static.developers.redhat.com/che/theia_artifacts/che.103ff3fa440e96277bc7.js is being linked. @davidfestal do you have any insight? Perhaps it's related to CDN support in Theia?
The reason pods are not staying in a running state is because Che is killing them after a timeout. Once the pods are running (which is happening because you're getting logs from them), Che checks the ingress for Theia for an http response before marking the workspace as "started". It looks like this step is failing (the Theia server is up and running but Che can't see that).
It looks like Che doesn't log where it's trying to connect for the server readiness check, even in debug mode, which is unfortunate. One way to test this would be to
- Get the URL of the ingress created for Theia (e.g.
serverp12xsau8-go-cli-server-8080.fqdnif I read the above correctly) while the workspace is starting. - SSH into the Che server container and try to curl that URL, looking for errors.
amisevsk
on 12 Nov 2019
@amisevsk @sleshchenko @davidfestal
May be the readiness check depends on that all content are available ?
I Reconfigured Che server and che workspaces to use an entreprise proxy ( for testing) The problem was the same, but when I provide an internet access to the client machine, the machine where I have my web browser that I use to access Che, in this case, I got access to Theia editor!
So my question is : how Can I build che-theia to work on an air-gapped environment ?
Thank you
NM4
on 18 Nov 2019
I opened another issue about Che Offline #15382
NM4
on 9 Dec 2019
Related issues
vanzhiganov
路
3Comments
sudheerherle
路
3Comments
JamesDrummond
路
3Comments
InterestedInTechAndCake
路
3Comments
l0rd
路
3Comments