Che: Suggestions for improving `eclipse/che start` on a host with SELinux enabled

[gbraad]

Running docker run -v /var/run/docker.sock:/var/run/docker.sock eclipse/che start on my Fedora 25 installation fails. It does not matter if the /var/run/docker.sock has been assigned to the docker group, of given chmod 777...

Reproduction Steps:

1. On Fedora 25, using docker, run docker run -v /var/run/docker.sock:/var/run/docker.sock eclipse/che start

Welcome to Eclipse Che

You are missing a mandatory parameter:
   1. Mount 'docker.sock' for accessing Docker with unix sockets.
   2. Or, set DOCKER_HOST to Docker's location (unix or tcp).
...

Che version:
5.5.0.

OS and version:
Fedora 25 (updated)

Docker version:

Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-4.gitf499e8b.fc25.x86_64
 Go version:      go1.7.4
 Git commit:      f499e8b/1.12.6
 Built:           Fri Jan 13 11:03:22 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-4.gitf499e8b.fc25.x86_64
 Go version:      go1.7.4
 Git commit:      f499e8b/1.12.6
 Built:           Fri Jan 13 11:03:22 2017
 OS/Arch:         linux/amd64

Solution:
The Che container needs to be run with --privileged, else docker ps within the Che container will fail on: /scripts/base/startup_02_pre_docker.sh, init_check_docker(), L289 ref

Although inside the Che container:

bash-4.3# ls -al /var/run/docker.sock
srw-rw---- 1 root 1000 0 Mar 17 08:20 /var/run/docker.sock

access to the socket will still result in Permission denied.

When started as:

$ docker run --privileged -v /var/run/docker.sock:/var/run/docker.sock eclipse/che start

Bound 'eclipse/che' to 'eclipse/che:5.5.0'
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
WARNING: bridge-nf-call-ip6tables is disabled
Welcome to Eclipse Che!

We could not detect a location to save data.
Volume mount a local directory to ':/data'.
...

allows this check to pass. However, successive interaction with the data volume will also fail.

Cause:
Likely caused by SELinux. If so, setenforce 0 (disable SELinux) should be described as part of the installation instructions or start banner. Better would be to describe steps that allow to function WITH SELinux enabled.

[TylerJewell]

Please do a search on fedora in the issue archives. I think this issue has been tripped into a couple times before with simple solutions. I thought our docs covered the bases on these items relating to selinux.

[gbraad]

https://github.com/eclipse/che/issues/4020#issuecomment-279431125

The reason for mentioning is mostly the fact that

$ docker run eclipse/che start

will not result in a workable environment on Fedora/CentOS. It needs a lot more prerequisite settings and changes before it works. It would be great to collect the information to make it work without having to disable SELinux... and then automate this for deployment

[ghost]

We could not detect a location to save data.
Volume mount a local directory to ':/data'.

It's because you have to mount some local dir to /data. So, an additional volume binding is mandatory in the start syntax.

[TylerJewell]

head slap - thanks @eivantsov - very obvious. I just did not read the error all the way down.

[gbraad]

@eivantsov Even with the data volume is not starting as expected. SELinux needs to be disabled else copy actions on the data volume will fail.

Eventually as @l0rd described

$ mkdir -p /var/chedata
$ docker run -p 8080:8080 \
       --name che \
       -v /var/run/docker.sock:/var/run/docker.sock \
       -v /var/chedata:/data:Z -e CHE_HOST=10.0.21.42 -e CHE_DOCKER_IP_EXTERNAL=10.0.21.42 \
       -e CHE_DOCKER_SERVER__EVALUATION__STRATEGY=docker-local --security-opt label:disable \
       rhche/che-server  # eclipse/che-server

starts and allows the Dashboard to be opened. But all the described issues with docker run eclipse/che start exist: "permission denied on socket and folders", "WS not detected", and so on...

[ghost]

@gbraad What error do you see when starting a workspace?

[ghost]

@gbraad also, use eclipse/che image

[gbraad]

I ended up with getting errors, such as:

INFO: (che start): Preflight checks
mem (1.5 GiB): [OK]
disk (100 MB): [OK]
port 8080 (http): [AVAILABLE]
conn (browser => ws): [NOT OK]
conn (server => ws): [NOT OK]

and the recommended way forward 'info network' IIRC, does not really provide a solution besides just another viewpoint on the misconfigured network. Instructions need to be more solution driven.

Note: this was just one of many. Therefore I tried to run as che-server, which worked with the afore mentioned settings. Will take some time later to check what went wrong with eclipse/che start. I first wanted a working situation, and then see what went/goes wrong with the recommended way.

In short, I personally dislike(d) the experience of using eclipse/che start (or codenvy/cli start), as it ended up being like one of those old adventure games; one mistake and you die... progressing by trial and error. each time refining your command does not make it interactive.

[ghost]

@gbraad I kind of agree that providing instructions that will fail and give you clues on how to fix mistakes isn't the best way to welcome a 1st time user.

As to your errors, I think the problem is in networking. CLI check fails. Below is the test that fails. Can you try running it manually?

docker run -d -p 32765:80 nginx
docker run -ti appropriate/curl sh
once in a container:
curl -v $IP:32765

You should get nginx welcome page content. If you cannot, then firevall blocks communications between containers. $IP is your VM/machine eth0 IP.

[gbraad]

@eivantsov Actually, the result is <h1>Welcome to nginx!</h1>. There is no firewall blocking internal traffic...

[gbraad]

@eivantsov is there a spec or description of what the start/introduction flow should look like? without having to dive into a lot of shell scripts under /scripts/. I would like to see how this can be improved.

[TylerJewell]

Start by studying:

1. https://github.com/eclipse/che/blob/master/dockerfiles/base/scripts/base/startup_01_init.sh#L243
2. https://www.eclipse.org/che/docs/setup/cli/index.html#cli-development

Recognize that key design goals are:

1. Offer the simplest starting syntax with as much guided help as possible so that people when they make syntax errors can navigate their way out.

1. Recognize that the entire structure of the CLI is designed to support custom CLI assemblies, so the Che CLI and Codenvy CLI are inheritance of the base image with extension points well defined.

2. We believe in mutability of the configuration, so the CLI is just a stateless image whose job is to verify that the environment is suitable for execution and then to use Puppet to generate a runtime template which can be used to run the system.

3. We wanted identical syntax for different assemblies so the end user syntax for codenvy and che are largely identical even though one is just a single server and another is a complex compose services with zookeeper and swarm.

4. Offline mode for enterprises is essential - there has to be a minimal and consistent set of steps where an admin can download everything needed to start the system offline.

5. We have certain rules baked in for version compatibility with "latest" and "nightly" as special case versions that get certain types of treaments.

[gbraad]

Example:

$ docker run eclipse/che start
INFO: Welcome to Eclipse Che!
INFO:
INFO: You are missing a mandatory parameter:
INFO:    1. Mount 'docker.sock' for accessing Docker with unix sockets.
INFO:    2. Or, set DOCKER_HOST to Docker's location (unix or tcp).
INFO:
INFO: Mount Syntax:
INFO:    Start with 'docker run -it --rm -v /var/run/docker.sock:/var/run/docker                                     .sock' ...
INFO:
INFO: DOCKER_HOST Syntax:
INFO:    Start with 'docker run -it --rm -e DOCKER_HOST=<daemon-location> ...'
INFO:
INFO: Possible root causes:
INFO:    1. Your admin has not granted permissions to /var/run/docker.sock.
INFO:    2. You passed '--user uid:gid' with bad values.
INFO:    3. Your firewall is blocking TCP ports for accessing Docker daemon.

User realizes that he might have to add the configuration to the docker.sock. and so, he adds tihis:

$ docker run -v /var/run/docker.sock:/var/run/docker.sock eclipse/che start
INFO: Welcome to Eclipse Che!
INFO:
INFO: You are missing a mandatory parameter:
INFO:    1. Mount 'docker.sock' for accessing Docker with unix sockets.
INFO:    2. Or, set DOCKER_HOST to Docker's location (unix or tcp).
INFO:
INFO: Mount Syntax:
INFO:    Start with 'docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock' ...
INFO:
INFO: DOCKER_HOST Syntax:
INFO:    Start with 'docker run -it --rm -e DOCKER_HOST=<daemon-location> ...'
INFO:
INFO: Possible root causes:
INFO:    1. Your admin has not granted permissions to /var/run/docker.sock.
INFO:    2. You passed '--user uid:gid' with bad values.
INFO:    3. Your firewall is blocking TCP ports for accessing Docker daemon.

user tries locally: docker ps... tries as a user docker ps, and notices the user has not been granted access. Sure. he modifies the permission on the socket and chmod 770 /var/run/docker.sock,newgrp docker and tries again... and command succeeds.

He runs the same command again: docker -v /var/run/docker.sock:/var/run/docker.sock' eclipse/che start...

same results... so he starts wondering if he really had to type docker -it --rm -v /var/run/docker.sock:/var/run/docker.sock' eclipse/che start ? no change in message and no actual error message follows. In this case, showing the error message would be a better idea. WDYT?

A simple test like

[ -S /var/run/docker.sock ] && echo OK || echo FAIL

around https://github.com/eclipse/che/blob/master/dockerfiles/base/scripts/base/startup_02_pre_docker.sh#L289

would have helped to check the state of /var/run/docker.sock. In that case you know something would be wrong with the socket. An additional test, such as:

ls -d -- /var/run/docker.sock > /dev/null

would give you a permission denied (exitcode 2) or pass (exitcode 0), you could accordingly change the message to indicate that a permission denied occured on the docker.sock.

At the moment this is left up to guess from the user. captured in:

INFO: Possible root causes:
INFO:    1. Your admin has not granted permissions to /var/run/docker.sock.
INFO:    2. You passed '--user uid:gid' with bad values.
INFO:    3. Your firewall is blocking TCP ports for accessing Docker daemon.

In this case we surely would know it is:

INFO:    1. Your admin has not granted permissions to /var/run/docker.sock.

[TylerJewell]

Those are definitely some fantastic suggestions. Would very much like to see those added.

Yes, I completely agree that we are not doing nearly enough in the types of tests related to the Docker daemon:

1. If the unix socket, there are a variety of additional tests that you can run like you just said.
2. If DOCKER_HOST with tcp, then also adding in TLS option for connectivity and appropriate tests.
3. We don't do anything to test systemd connectivity over socket right now.

[gbraad]

as it is a permission error and likely an admin... So perhaps it is in place to mention the situations related to AppArmor and SELinux.

Running as:

$ docker -v /var/run/docker.sock:/var/run/docker.sock:Z eclipse/che start

would have passed in case of SELinux. So a suggestion that is specific to the permission error seems appropriate.

[TylerJewell]

When will we see the PR?

If you submit one, I'll test on mac & windows.

[TylerJewell]

BTW, what is the -S test on a file do? I cannot seem to find any reference to it with some quick google searches.

[gbraad]

@TylerJewell It tests for socket, but it is bash specific.

[gbraad]

Note: by this time the user has also generated several stale containers by successively running

$ docker eclipse/che start

instead of

$ docker --rm eclipse/che start

[gbraad]

@TylerJewell http://tldp.org/LDP/abs/html/fto.html

[TylerJewell]

We played with auto-destruction of old images previously and always got into problems with it. There are some non-mutable problems if you try to have a container clean itself up when it finishes.

[gbraad]

will try to get a PR up soon, but would like to test for Windows myself first. Any pointers on how to get this going?

[TylerJewell]

We do our tests on Windows 10 with docker for windows. You can do it on a boot2docker VM too - the results are largely the same. They are both linux VMs.

[gbraad]

OK... will also run it against Minishift, but this actually always failed for the port available checks.

[TylerJewell]

On the minishift, you should connect with the redhat openshift team that has been doing the port of che onto openshift. they have it entirely working now in an openshift branch. The CLI needs to be extended so that we generate a minishfit deployment configuration.

Each run of "config" command of the CLI should generate three things:
docker-compose.yml for running che on the host
docker-compose-container.yml for starting che from within a container
docker-compose-minishift.yml - a custom docker compose file used to start a minishift cluster that will have che deployed within it.

[gbraad]

@TylerJewell I am on the Minishift team, but in this case I just meant re-using the docker environment inside Minishift (not even OpenShift itself yet), Already hooking up with the guys working on this...

[TylerJewell]

Your ambitions are not big enough :)

Seriously, with the openshift branch che is entirely functional inside of openshift now. They will be demoing it in many ways for upcoming DevNation. So if someone can get us some host scripts on how to start minishift with a pre-packaged pod that contains the che branch built for openshift, we can probably fairly easily extend the CLI to support starting minishift with che pre-installed.

[gbraad]

Your ambitions are not big enough :)

Understanding the startup process and deployment strategies are key, before dumping it all inside Minishift. I'd rather not hope for the best. But yes... this would be ideal and a goal. An earlier attempt of running che on Minishift is here: https://github.com/l0rd/openche, but is not up-to-date with the current images.

[TylerJewell]

that was mario's work as a test to understand what was need to create a proper openshift connector within Che.

We really need two different modes for minishift:

1. Assuming an existing minishift cluster and admin credentials, deploy che onto it.
2. A secondary packaging which is minishift + che prepackaged together and started on the host, probably only using virtualbox or hyperv.

The che deployment model is a single che instance for each user and each che instance manages its own workspaces. Eventually when codenvy runs natively on open shift, it will have a different deployment model.

[gbraad]

@TylerJewell The second step is on the roadmap as a general functionality, as we would like to prepackage different kind of containers/environment or usage profiles. Not sure in what timeframe we are able to deliver this... Note: At the moment we are primarily targeting Hyper-V on Windows (but Virtualbox as fallback). Already started to conversation with Mario (since yesterday) (-:

[TylerJewell]

Will the hyperv packaging of running minishift interfere with people who already have docker for windows with a moby VM installed?

[gbraad]

No interference

However, the trick is to redirect the calls correctly if you want to interact with the Docker daemon. This is why I actually also looked into the use-case of just re-using Minishift's docker to run Che:

$ minishift start \
  --cpus 2 \
  --memory 4096 \
  --iso-url=https://github.com/minishift/minishift-centos-iso/releases/download/v1.0.0-rc.3/minishift-centos7.iso
$ minishift ssh "mkdir -p /mnt/sda1/chedata"
# alias given purely as example. you can actually do:
#   powershell: & minishift docker-env | Invoke-Expression
#   bash: eval $(minishift docker-env)
$ alias minidocker="docker -H tcp://`minishift ip`:2376 \
  --tls --tlscacert=$PWD/.minishift/certs/ca.pem
  --tlscert=$PWD/.minishift/certs/cert.pem
  --tlskey=$PWD/.minishift/certs/key.pem"
$ minidocker run \
  --name che \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /mnt/sda1/chedata:/data:Z \
  -e CHE_DOCKER_SERVER__EVALUATION__STRATEGY=docker-local \
  --security-opt label:disable \
  eclipse/che start

In this case, we are providing a possible alternative for D4W, which is actually based on a CentOS image with SELinux enabled

Although you shouldn't have to, as oc as client would be wired up correctly after deploying the instance.

[pjgoodall]

I am having the same problem on Ubuntu 16.04 LTS - which is not an unusual platform.

• Docker version 17.06.1-ce, build 874a737
• Ubuntu 16.04.3 LTS

If the Eclipse Che website says "Run Eclipse Che now with: docker run eclipse/che start" then it must work

[garagatyi]

@gbraad Can you provide the command line that was able to start Che properly on your system?

[coolbrg]

@garagatyi ,

$ docker run -p 8080:8080 \
       --name che \
       -v /var/run/docker.sock:/var/run/docker.sock \
       -v /var/chedata:/data:Z -e CHE_HOST=<machine-ip> -e CHE_DOCKER_IP_EXTERNAL=<machine-ip> \
       -e CHE_DOCKER_SERVER__EVALUATION__STRATEGY=docker-local --security-opt label:disable \
       rhche/che-server  # eclipse/che-server

[gbraad]

@budhrg already posted it, but @garagatyi this is the command I used that time:

• http://gbraad.nl/gists/gists/aa5273d7021f724560dd8b5e0cafd100/
• http://gbraad.nl/gists/gists/18607310e0b122eb3790558e6f4d0795/

We are currently working on an alternative solution with Minishift. This can be tracked at: https://github.com/minishift/minishift/issues/1264

[garagatyi]

@gbraad but in this comment you said that it was not enough, did I get it wrong?

But all the described issues with docker run eclipse/che start exist: "permission denied on socket and folders", "WS not detected", and so on...

[gbraad]

This got fixed along the way in another PR

[garagatyi]

@gbraad thank you!

@pjgoodall can you elaborate on what exactly doesn't work in your case?

[gbraad]

@pjgoodall I would suggest to create a new issue, as the original issue I filed was releated to SELinux and how some of the checks are performed.

[pjgoodall]

Thanks all. I’m no longer an active user of eclipse che. So have no context.